Comment Re:Completely different contract/machine/goals (Score 1) 99
The bulk of the new blue waters will be pure opteron nodes, with only 35 of the 270ish cabinets using GPUs. They obviously are assuming that most users will be running with only x86 cores. They ordered a few dozen cabs of GPUs, probably with an eye to where the industry will be heading over the lifetime of the machine, not where the users are today.
It's true that interlagos cores are a poor competitor to power7 core to core. However, they fair much better if you use the entire module. Think of interlagos as an 8-core processor with 2 threads per core, and all of a sudden it looks a lot better. Power7 is probably still better, but at ten times the cost.
Furthermore, just because a single node of power7 is an awesome node, does not mean that a many-thousand node supercomputer, composed of those nodes, is also awesome. If the IBM blue waters machine were just about the processors, they would have clustered together a bunch of bladecenter704s. They would not have bothered to bid the p775 system. If you want really fat SMP nodes, then they would have bid a bunch of p795s. Obviously they tried to make a really high-bandwidth shared-memory interconnect for the p775, and they failed. Either it didn't work reliably, wasn't fast enough, or cost too much. IBM didn't step away from the deal because the clock speed was 10% low, or because their costs rose enough to make their margin slim. You don't screw a customer in such a high-profile way unless you're going to lose a LOT of money on the deal.
I notice, looking at the top500 list, that IBM managed to sell the 10 cabinets of p775 that they built for NCSA. A few weather forecasting sites in canada and the UK are running 5 systems at 2 cabinets each. Tellingly, a couple of those customers have a pair of 2 cab systems, rather than a single 4 cab system. That tells me that the interconnect isn't scaling to larger numbers of nodes. A problem for a really big system like blue waters.