Google isn't detecting potholes? Back in 1985, we had that on our DARPA Grand Challenge vehicle. The LIDAR on top of the vehicle was generating a ground profile. This was for off-road driving, where that's essential. I'd assumed Google was doing that; they have a Velodyne laser scanner that provides enough information.
In traffic, sometimes you can't see a pothole because it's obscured by a vehicle ahead, but if the vehicle ahead doesn't change speed, direction, or attitude, it's probably safe to proceed over the ground it just covered. On high speed roads, you can't see distant potholes clearly because the angle is unfavorable, but if the road ahead looks like the near road, and the near road profiles OK with the LIDAR, the far road is probably good. That's what the Stanford team used to out-drive their LIDAR range. (We didn't do that and were limited to 17MPH).
Fixed road components should be handleable. People, bicycles, and animals are tough.