Once we're stepping out of the realm of advanced cruise control and into active driving, it will clash even if they don't want to take responsibility. "I didn't expect my car to make the turn and fail to yield, you can't expect me to undo every mistake" "I saw it coming and could brake down, but my car didn't realize and speeded up and caused the accident" "I tried to hit the ditch and avoid those school kids but my car refused to go off the road, running them over."
And once you've seen the computer do a maneuver 99 times you'll assume it will the 100th too, even if it's got some kind of sensor glitch meaning that no it won't. It's one thing to see a situation in front of you and slam the brakes in a duel with the computer, but then you'd have to co-drive all the time. It's another thing entirely to see a situation in front of you, wait a few moments, realize oh shit the computer isn't going to do anything then hit the brakes.
And one thing that's important to remember is that accidents are generally not legally punishable, the driving must be negligent or reckless for that and being surprised and acting panicked is legal for a human driver. If the car is going the posted limit, obeying traffic regulations and hits the brakes it may meet the legal minimum even if there will be an accident and the result might be sub-optimal, unless we hold autonomous cars to a higher standard.
As the backup driver I'm certainly only human, my reaction could certainly be surprised and panicked. To roll that liability past the car and onto me there must have been some rather damn obvious reason why I had to intervene. It would have to be reckless or negligent of me to think the car can handle it better than me. If it ever got to court I'd argue that's just not true, in perfect hindsight maybe it was a poor split-second judgement but that's not a crime.
Or the TL;DR version: I doubt you'll ever be held legally liable for not taking over control, that you might or possibly should have yes but not that it was reckless or negligent not to. So in practice I think distracted driving will be legal, if not in theory.