Yes, because the terrible drivers we have all seen are not causing accidents as it is.
Every time I see a discussion about autonomous cars, someone chimes in that there are terrible human-driven cars on the road, and that an autonomous vehicle cannot deal with that. What they fail to mention is that no human drivers can really deal with them either, if the terrible driver is driving so badly that an accident is bound to happen.
Bad drivers causing accidents because they are on the cell phone? Yep. Bad drivers causing accidents because their vehicles are not properly maintained? Yep. Bad drivers causing accidents because the do something unpredictable without looking? Yep.
In all of those cases, you are right - an autonomous car will probably be no better than a human driver. You have yet to convince me that it will be *worse*, though. As more autonomous vehicles are on the road, though, accidents caused by bad, distracted drivers will go down. So at worse it is no improvement, with an almost assured big improvement as time goes on. Or we can just stay with the status quo.
You are right, though - replacing the entire infrastructure is not going to happen. I am guessing the Vehicle to Infrastructure communication they are talking about are things like red lights (why have a camera in the car to determine the color of the light when the intersection can just broadcast directly to vehicles?), train crossings, and so on. An autonomous vehicle should be able to deal with these things as they currently are, but if a town's red lights are due for replacement, why not replace them with autonomous vehicle friendly versions?