I agree with what you said pretty much.
I don't like driving at all and if someone gave me a Porsche, Ferrari, Lamborghini etc I would sell it and if I could get a self driving car I would otherwise I would just save the money for now. I look at cars as a way to get from point A to point B and I don't want to be bothered in any way by them.
What I want is a safe self-driving electric car that can take itself to the repair shop or call for help when needed and arrange a replacement so I can worry about other things.
The only problem is that insurance is based on risk pools. This means that as people switch to self driving cars the risk pool for cars that people drive shrinks and by definition they are the most unsafe drivers compared to the autodrive cars. This will mean insurance will go up and move people will stop driving their cars for money reasons and the insurance will keep going up.
Eventually very few people will be able to drive their own cars no matter what their views on on it since they won't be able to afford the insurance. Not sure if I really like that endpoint very much but it would end up with a much safer world and much faster transport.
Maybe there should be more tracks and designated areas for people to drive for themselves or something.