The number of problems caused by autonomous cars will be inversely proportional to the number on the road.
Respectfully, I don't think you pay close attention to all the little "hiccups" that occur during daily driving - no one does, because our brains handle them with ease. As a old firmware guy, I know digital computers won't be able to do this because there are too many variables, forever changing. When you drive from now on, imagine you're blind, and have perfectly memorized the road and could drive it with no sight. In the future, look to see what alters your path during your commute (or what has changed since last time), and then think about what the car would have to know to handle the situation without slowing down. In other words, if you didn't have to slow while you were driving, then the AV shouldn't either, because it's better than we are, right?
It will never (instantly) know the difference between a very small ball rolling into the street ("Where's the child?") and an oak leaf. A small tree branch versus a live wire (I saw this in the street once after a major storm), where, one I can roll over, the other, not so much.
There will be a critical mass beyond which insurance companies will begin charging extravagant fees for a manually operated vehicle.
I doubt that'll happen because: A) AVs will always be more expensive than human-driven cars. Some people can only afford the bare minimum. B) Some people enjoy driving fun cars and there won't be AV versions of those. C) Many people own classic cars (like me) where those will never be "AVed". D) There is enough voter/buyer mass of A, B & C to where insurance will not be significantly higher for non-AV drivers. E) Semi-trucks parking and weaving through tight industrial parks is an art form. Pulling that off with sensors is problematic at best, especially when trailers are universally interchangeable and have no electronics. (Automating all trucks would be a thief's paradise.)
Issues with reading signs are a non starter. Once adoption begins to pick up you will quickly see digital information systems added to existing road signs. All of this tech exist right now and most of it is mature. It just hasn't been put together yet.
There are an order of magnitude more Podunk towns than metropolises; they will never be part of this digital information system. They might change a sign and then have to report it somewhere? I don't think they'll bother - some have no full time staff (I sometimes call on the rural Midwest).
In about 20 years people will be complaining about how manual drivers are always causing accidents and issues with traffic flow.
Is this before or after the flying cars that are just 5 more years away? I'm not being snarky at you, just the idea that tech is easily mass-distributed, and can solve all.
"The power of accurate observation is commonly called cynicism by those who have not got it." - George Bernard Shaw
"The vast majority of successful major crimes against property are perpetrated by individuals abusing positions of trust." -- Lawrence Dalzell