I'd say never. Why? Because driving on random roads and locations in varying conditions requires *intelligence*.
Most attempts at computer-driven cars so far basically just put cars on an electronic track and throw in a little bit of sensor data and pre-programmed logic.
Yes, 99.9% of driving is just dull "stay between the lines" but it's that .1% of unexpected shit that requires a brain. (not that too many people these days have those)
What happens when some crap falls off a truck and the sensors don't quite pick it up? What happens when someone runs out in the road in front of it? Can it tell the difference between a rectangular dark black flat piece of dangerous metal on the road and dark shadow from an overhead roadway sign? Would a computer even realize you might need to drive off the road in the event of an emergency or special circumstances? There are an almost infinite number of possibilities that you just can't pre-program in or plan for.
Throwing control back to the user probably wouldn't work well in practice, because once a car can drive itself sometimes, people will expect it to drive itself ALL the time.
And when something does go wrong, who will people sue? Right now it is the individual driver of the vehicle. I think car manufacturers would prefer to avoid the risk of being sued and leave responsibility where it is.
So what if someone creates a computer that is really intelligent enough to do that? Well, I'd suggest getting out of its way as it kills all humans. :P