Usually they keep the Newest-1 machine around for a little while so that users can slowly migrate over to the new machine. Even with subsequent generations of supercomputers from the same manufacturer, there are often little difference that make migration tricky. Once most users have migrated, they generally decommission the machine and scrap it. Used machines just aren't worth running. They are big and heavy to move, require so much power that you need lots of expensive electrical work to power, and HVAC installation to cool. You also need highly trained administrators to operate. The cost of the hardware is only a small fraction of the cost of having a supercomputer. If someone is going to pay all those other costs, they are going to want to get a lot of performance in return. Old machines just aren't going to justify the expense.
Sometimes they can get the manufacturer to take it away, and a few parts get recycled. Cables, in particular, often are still useful from one generation to the next. You could ebay the processors or RAM, but they're generally 6 years old by that point, and not worth anything. Frames get sold for scrap steel. Cables for scrap copper. ICs generally end up in a landfill.
Back in the 80's and 90's old supercomputers became museum pieces, but modern supers are all just rectangular boxes. Not really inspiring nostalgia the way a Cray XMP might. There also used to be a lot of gold on circuit boards that was recovered, but those days are long gone.