Along the same lines, it doesn't seem difficult to take control of the system while it's actively driving. It's not hard to disengage cruise controls or stop a car using Park Assist or Lane Assist from turning into something not seen by the sensor system. Why is it hard for me to grab the wheel from the "hands" of the auto-pilot in the Google car?
Here's the best example I can think of -- let's say you are the understudy for a radio actor with narcolepsy. You both have the script, you the understudy are following along word for word as the actor is performing. Suddenly the actor falls asleep and the words stop. How many seconds pass before you pick up where he left off? You are as aware and able as you can possibly be without actually anticipating something you can't anticipate, and I believe it would still take me a few seconds to switch myself from simply paying attention to audibly reading words.
Second scenario is the same except that you are both in sound booths reading the words and the actor is the only one with a hot mic. I believe it would be faster for me to be reading aloud with the actor and trigger my mic to go live at the necessary time -- however in doing so I am saving no effort over doing all the reading myself in the first place, so the application to automated vehicles is somewhat limited.