"So at some point in the future we will see that computers achieve a higher safety level than any sample of human drivers, while remaining imperfect. At this point, it will probably become necessary to ban manual driving on highways, for the protection of the other people on the highway."
Simplistic. For one thing, automatic things like cars is surely going to turn out to be MUCH harder than most anyone expects. Ask folks who have experience applying computers to real world physical processes. The number and variety of things that can and will go wrong is astonishing. Especially once one ventures of the expressway and away from pre-plotted test circuits to a world of kids, pets, horsecarts, wildlife, poorly marked intersections, bicyclists, construction projects, etc, etc, etc. Moreover, even after most of the bugs are ironed out, there are going to be occasional situations -- a road covered with heavy snow (where's the edge?), a fireman or policeman directing traffic up over the sidewalk and up a one-way street in the wrong direction -- that will require human intervention.
(And removing the steering wheel and pedals is almost certainly a bad idea. They'll give us some sort of probably horribly designed joystick or touch screen interface instead?)
I would agree that there may well eventually come a time when overriding the computer -- at least in urban areas -- will be something that can require an explanation after the event. But that time is surely a LONG way in the future.