Agree too, but you've missed a couple factors that should be considered.
1. All the "You can have my steering wheel when you pry it from my cold, dead fingers!" people.
This problem will almost entirely be solved by the pricing of insurance and cars. If that doesn't resolve any problems then drivers licenses and regulations will solve the rest. Consider that once the percentage of cars that operate autonomously hits a tipping point then the drivers that are not represent a hazard and remove the possibility of many of the benefits of driverless cars. Costs of insuring a car to drive it by oneself will go through the roof due to the much higher accident rate and most people will not be willing or able to pay that. That higher insurance cost alone will greatly reduce the number of people driving their own cars. Then if people driving their own cars is still a problem licenses can either no longer be issued or driving simply not be allowed.
I was once like you I loved the freedom of driving, but I've become more flexible and realized that as long as I can get where I want to it doesn't matter how I get there. In fact I'd rather save the time driving for something more enjoyable. I get the same sense of freedom by taking a long distance bike journey.
But I think you're spot on with a lot of the potential risks of trusting the driverless cars. The extent to which those are managed well will determine whether driverless cars represent a benefit or harm to society.