The real value of a self-driving car is just that, fully self-driving.
It's having something that can drive while you're asleep, reading, or maybe even working on your laptop.
It's something that can drive your 10 year old to school, drop them off, and then drive back to the house so that other people in the household can use the car.
And just as importantly, it's something that someone who is not fit to drive-maybe for medical reasons, maybe because they have not slept in 24 hours, maybe because they are drunk-can use to safely get where they need to go.
So no, the danger of salf-driving cars isn't that people will decide not to be in the driver's seat, the danger is that both automakers and regulators will try and give us supposedly self-driving cars that can't handle those cases, and then be surprised when things go horribly wrong, or when people just don't see the value in buying one.
Personally, I plan on ownning a true self-driving car very soon after I can buy one that can do the driving when I can't, and I bet that the vast majority of legally blind adults with enouh money will be right along there with me. But that won't happen anytime soon when people are acting like you need a driver for it to be safe.