You think with the popularity of programs like Top Gear that there won't be millions around who still prefer to drive their own cars?
I think that sooner or later they'd be restricted to the track for their self-driving desires(and I don't think motorsports will go away that soon) due to insurance costs.
If self driving cars are shown to reduce accidents by 90%, that's roughly a 90% reduction in insurance expenses, which can amount to a couple thousand a year pretty quick. In short, even if a self driving car is $10k more expensive, if it saves you $1500 or more a year in insurance costs, discounting any savings from improved fuel economy or time recovered, it's worth it.
Now consider who has the highest insurance costs - people with DUIs - I can see drunk drivers being forced into self-driving vehicles very quickly, without manual overrides. People with bad driving records. Young/New Drivers.
Then you get the exact same thing as you did with automatic transmissions. Once you start putting those that would be driving in automated cars rather than making them actually do the driving, they'll tend to stick with self driving cars. Then it'll expand to the point that finding a vehicle with manual controls is about as easy as finding a vehicle in the USA with a manual transmission.