This is one of the many reasons why I won't be an adopter, or at least an early adopter.
If I have to babysit the car second-to-second, then there's little point in having the car.
And I won't trust the car to not malfunction and kill me or someone else, so chances are I will be monitoring it. Maybe the car companies / Google / whoever will say "trust me, you can sleep in the back seat" but I'll have a hard time accepting that for at least another 15 years.
And lastly... I trust myself as a driver (never had an accident). To ME, my skills are a known commodity. And if I mess up, then I messed up and can own up to it. If my car messes up and I wasn't babysitting it enough or some birds**t landed on a camera, then if I'm liable then that sucks. I can accept that I made a mistake and pay the consequences easier than something I own made a mistake and now I have to pay for its consequences.