Driving on the road isn't the problem, it's driving on the road and not hitting the deer that just ran into it, or avoiding the knucklehead who just swerved into your lane because he's drunk.
Actually I think that avoiding unexpected road hazards will be the self-driving car's strong point. A car can be watching in all directions, all the time, and can react within a few milliseconds to avoid a collision. Computing the best way to react is not a terribly difficult problem either, since all you need is a reasonable physics model in which the car can play out the likely results of each of its various options, and then choose the option that looks like it will yield the best result. (if you want to imagine what driving is like for a computer, imagine that time was slowed down by a factor of 1,000. You'd find that driving was more like chess and less like an arcade game)
I think the difficult parts for a self-driving car will be the parts involving communication with other human beings -- e.g. noticing that the traffic cop in the intersection is signaling that the car should stop (or go) now, or that the road cones placed between two lanes are meant to indicate that the right lane is currently closed to traffic. Compared to that, getting collision-avoidance-physics right will be fairly straightforward.