Self-driving cars not only use a variety of sensors to assess the environment, but also have systems based on algorithms like SLAM (Simultaneous localization and mapping) to help them position themselves relative to the environment, and position landmarks relative to them. All kinds of sensors are involved, but especially laser range sensors which would prevent the kind of problems caused by GPS returning invalid results (the car won't just drive into a wall, it will avoid the wall and reduce the belief associated with the current location). GPS is just an extra sensor, not a bunch of set-in-stone instructions.
They don't hit the pot hole because there are computer vision systems that, along with the range sensors, can make a reliable guess at whether that is a pot hole or not, and avoid it. Speed would be irrelevant as the computer can react faster, and more accurately than a human driver could.
When it comes to this kinds of algorithms, sometimes they are *too* efficient, and you have to route around that: a good example is going around walls, in which the car might decide to hug the wall and take a turn very, very close to the corner - but this is not optimal as a) the driver would probably freak out b) Movement and location sensors are not perfect, you always have to consider actuator and sensor noise. So, the algorithms are complemented by a penalty for getting too close to objects, even if it wouldn't cause a collision.
I hope that helps paint a broad picture of the system to make a bit more understandable.
