This Google experiment proves that you can build a self driving car who can drive safely for thousands of miles on actual public roads. Yes, there are some additional conditions which are not practical (like the driving the path manually before to record a precise map), but it has been done.
Most drivers are not very much 'observing', I just read somewhere that the Blackberry outage caused vehicle accidents to drop by 20-40% in some Gulf states. Observing drivers, Ha !
Just compare how many accidents are caused by trees falling onto the street compared to texting.
The biggest problems to solve are the cultural, legal and liability issues.