The CRAY supercomputers are actually blade chassis, so you can do several generations of upgrades just by updating the blades. When we do have a to replace more components, the old parts are shipped back to CRAY for recycling. When the upgrade is more of a forklift upgrade, we will sometimes run them in parallel. This is what happened between Jaguar and Jaguar-PF. Jaguar was sent back to CRAY last year so that we can make room for whatever will replace Titan (the blade level upgrade of Jaguar-PF).
We have about 10 on-site CRAYons at all times at ORNL, and when we have a large upgrade process happening, extras are brought in.
At some point you do have to look at the Watts/Flop of the old systems, and you will get to a tipping point where it's simply too costly to continue running on the old equipment. When you're pulling down systems measured in Megawatts, you have to.