I guess your major misunderstanding is that the applications running on supercomputers could somehow be done in the (loosely coupled) way that Google does its data mining. Since you're a professional, too, please refer to this Wiki article on stencil codes, one of the major classes of codes that run on supercomputers. If you find a way (or at least a pseudo-code formulation) to transform these applications into loosely coupled codes, then I would not be the only one to be curious to hear about it. You'd transform the whole industry. In fact this is not possible, though.
But I agree that software will need to help with reliability and will have to actively manage node eviction/addition.
BTW: comparing Google and Cray is really like comparing apples and oranges: they're in different markets. The market for supercomputers is extremely small, the market for (online) advertising is gigantic.