Freeway driving is trivial: you don't hit what's in front of you, you don't hit what's beside you. Basic sensors can pull off both of these feats. You get bonus points if you can stay in a lane, but plenty of shitty human drivers manage to pull it off following those two basic rules.
Getting off the freeway is where it starts getting difficult. Even google maps sometimes misses the exit and tells me to turn right while I'm doing 60 over an overpass.
Freeway travel everywhere is pretty much the same.
On local streets, there are quirks.
However, this problem can also be solved by doing a Google street view style of predetermined intentions of how the roads were designed instead of computing them on the fly.
The traffic signals are also not standardized. The yellow in a 45mph road going downhill is shorter than the yellow at a 20mph road. Also, this behavior changes with time of day in some lights. So, when the light turns yellow, the car has to make a decision to keep going or brake for a stop.
So, here also the entire traffic light timings need to be recorded and stored to make right decisions.
Construction again changes things. But, they are easiest because they can be easily standardized.
Snow in Michigan changes things but I guess if you can automate the snow plows also, then it will make things easier for self-driving cars since the snow is plowed in some pre-defined method.
When a majority of the cars are self-driving, then the problem will be simpler. It is the phase when a few cars are self-driving and most are human drivers, then it becomes tricky.
But hopefully in the next decade, all the cars will be self-driving.
Perhaps all cars will be rentals in the future. You can have a car pick you up and drop you off and then go somewhere else. This will make car sizes smaller because people will not have to buy cars with the worst case use in mind. I see huge trucks with just a single driver on the freeway all the time. People will rent cars depending on passengers and how much stuff they have to move.