...and this is why I don't like the balloon analogy -- people point to the way in which the analogy is completely unlike the analogous system, and then assume that there is a comparable property in reality.
Think of the universe as the dolly zoom in the movie Vertigo. No matter where Jimmy Stewart is, as he climbs up the tower, when he looks down you see the bottom of the tower stretching away from him. There is no real center point -- it's not like the center of expansion suddenly moves up the tower with him. And it's not as if the bottom of the tower is expanding into the ground below. It's just that the space between him and the bottom appears to stretch away.