Cray XT-3 Ships 260
anzha writes "Cray's XT-3 has shipped. Using AMD's Opteron processor, it scales to a total of 30,580 CPUs. The starting price is $2 million for a 200 processor system. One of its strongest advantages over the std linux cluster is that it has an excellent interconnect built by Cray. Sandia National Labs and Oak Ridge National Labs are among the very first customers. Read more here."
How big is it? (Score:3, Interesting)
The article doesn't appear to mention its dimensions, and I'm curious to know what kind of space you need to install this baby. Anyone got any idea?
real FPU operations (Score:5, Interesting)
I ask, because I remember that the Athlons beat the pants off the Pentium 4's in FPU operations, so all the benchmarks were rewritten to use SSE2.
Heat... (Score:1, Interesting)
Opterons and PowerPC together (Score:5, Interesting)
The XT-3's biggest comptetitor in this segment must be the BlueGene/L type super computer made by IBM. The processors in Blue Gene/L is a custom built dual core version of the PowerPC 440 with built in high speed interconnects.
Just like IBM have a finger in all the future game consoles, they seem to have a finger in several of the next generation super computers also. Nice going IBM.
Intersting note (Score:3, Interesting)
This is the first time i see a shipped linux with this file system. Now the intersting part is that lusterfs is made for linux clusters, but this monster is not a cluster... any body can shed some light?
Re:imagine a... (Score:2, Interesting)
Re:real FPU operations (Score:5, Interesting)
1) You can use it in scalar mode, in which case it's almost like x87, only a bit faster because:
a) It doesn't use a braindead register model (stack)
b) On P4, you can do a mul and an add in parallel with SSE, but not with x87
2) You can use SSE intrinsics. It's not as easy as "normal" programming, but easier than assembly and almost the same speed.
3) Unaligned access is possible. It's slower than aligned access, but overall better than non-vectorized code.
4) Trig is so slow that SSE/x87 doesn't matter (unless you write approximations, in which case SSE will also be faster).
Re:cray (Score:5, Interesting)
Although it's true that Cray was not growing strongly before the SGI buy-out, it was not failing either. It could have kept running quite happily for many years, but in the bizarro-world of Wall Street, a company which is not growing is dying. I so love it when economists use biological terminology for corporations. In Wall Street's thinking, the only healthy growth would be a cancerous tumor.
Anyway....
The whole SGI-period of Cray is actually quite fascinating, and I suspect the true story will never be fully known. Lots of SGI engineers had their non-Cray technology branded with Cray marketting names, most egregiously LegoNet becoming CrayLink. Lots of Cray folks - aka. Crayons - felt that the core of their company was gutted by an SGI operation which didn't care for the extreme high-ends of HPC.
One rumor I heard, from a well-placed source, is that the Cray merger with SGI was primarily arranged by the USG. The intelligence services have huge investments in both company's products, so the merger between them made sense. I was told that as a quid-pro quo, the USG had an in-principle agreement to continue purchasing Cray gear to provide enough revenue inside SGI to keep both Cray architectures alive. However, certain parts of SGI felt that the US government didn't live up to their agreement, negotiations to rectify that weren't successful, and so SGI management defunded significant aspects of the Cray engineering work.
Also, FYI, Cray is one of those companies which will never totally go "belly up" anyway. Given the sensitivity of the work which they did, their support databases alone are full of sensitive and/or classified information. Should the company cease trading, it would be acquired by a shelf company whose sole function is to ensure this data would remain private. That's been the fate of almost all of the now-defunct supercomputer and high-end graphics companies who formerly supplied the defence and intelligence market.
Re:You don't have to begin to imagine (Score:5, Interesting)
More interesting is this spec:
Acoustical Noise Level: 75 dBa at 3.3 ft (1.0 m)
For comparison, that's roughly the same as an average vacuum cleaner when you're operating it, or maybe a good-sized pickup truck passing you in the next lane.
And remember, this value is *per cabinet*. You have to do a weighted sum over all the cabinets in an installation to get a true dB level. I wonder whether the maintenance people will have to use noise-level exposure limits for this baby.
And here I was, complaining about the quiet whine of my PC's fan.
Re:You don't have to begin to imagine (Score:5, Interesting)
that's amazing. how did the cray guys get a kilovolt-ampere that is not equal to a kilowatt? just goes to show you the power of fast interconnects.
Re:newfangled buzz. (Score:3, Interesting)
Re:Opterons and PowerPC together (Score:3, Interesting)
* first, sgi still makes and sells supercomputers, they are far from faded. they also own cray (or did).
* tandem, bought by compaq, we all know what happened there.
* hp sells a superdome once in a while. but nobody seems excited about their itanic systems.
* sun, rotting with their out of date cpus.
* fujitsu is doing well in the supercomputer market.
* nec is also successful.
* ibm, of course.
and you mentioned motorola? you're joking, i hope.
the largest purchasers of supercomputers in the world - national labs and the nsa, like to buy american hardware. they've always had a hand in keeping the industry afloat. notice that the big labs tend to round-robbin their supercomputer vendors so that they buy a machine from each vendor.
No, what stands in the way is price (Score:4, Interesting)
E.g., yeah, having a 30,000 CPU super-computer to simulate your gene model on would be nice. Forking over half a billion for it, well, it's suddenly not that nice any more.
Having one of those to simulate an electronic circuit, now that would probably rock. Again, paying half a billion for it, suddenly isn't that attractive.
The real question isn't how nice a toy you'd like to have, it's ROI. (Unless you work for the government, and just have a budget you _have_ to blow on stuff, whether you need that stuff or not.)
And in that context, you'd be surprised what you _can_ do with a lot less expensive toys.
Having Cray's custom interconnects sure is impressive, but for a lot of problems they're not even needed any more. _That_ is what killed Cray.
Most RL problems are not really the kind described as "_one_ huge indivisible data set, that you have to process in _one_ huge batch process." They're more like "we have this process with a small data set that we have to run 100,000,000 times." Most design problems or biology problems are really of that kind: run the same thing 100,000,000 times with different parameters.
And as Seti@Home or Folding@Home proved, a helluva lot of those don't really need _any_ kind of shared memory or fancy interconnects. The real ticket is noting that instead of accelerating the batch run 200 times, you could just split it into 200 smaller batches ran on 200 single-CPU machines.
The super-computer solution costs 2,000,000 just for the machine alone, while the 200 PCs solution costs 200,000 or so. I.e., 10 times cheaper. Better yet, the 200 PCs solution is also far cheaper to program. (Anyone can program a non-threaded batch app.) _And_ for that kind of a problem the 200 PCs solution would actually finish faster, since it has no contention issues whatsoever.
Again, that's what really killed Cray and the super-computers. They're techologically impressive, they're a geek's wet dream, but... for 99.9% of the problems out there they're just not worth the price any more.
... Back in my day .... young whippersnapper (Score:5, Interesting)
So come on, ante up. How many remember being awed at the mere sight of old Crays back in the day? Like the Cray-3? I remember the first time I saw a Cray .... thing was in an anti-static environment. To access it, one had to pass through an airlock and be "decharged" or "depolarized" etc. Basically they some how charged the air to get rid of static electricity. Then you had this system that was running *in* liquid! Take that "Oh I'm so cool cause I have a l337 haX0r water cooled CPU" overclockers
They (Cray) were so proud of this accomplishment that the upper portion of the cabinet was some kind of plexiglass so you could see the fluid as it moved, and moved wiring and what not with it. Very surreal feeling, almost like the thing was breathing.
And what about the Cray-1? Wasn't that a true testiment to 70's *art* and sculpture? The thing looks like some kind of freaky bus station bench with it's odd red and white panels and black base. Though, I don't know if they all looked like that, maybe you could get them in other colors?
Ahh .... those were the days.
Re:are you sure you remember seeing the Cray 3 ? (Score:3, Interesting)
Looked around on the net, as well as a couple other /.'rs here, and someone posted a link here to a 2 and I found a pic of a 2 with the waterfall system that was mentioned by another person, and I must accept defeat within the loosened strands of my unraveling mind.
It was indeed a Cray-2 that I remember so vividly. Nevertheless, still an extremely exotic machine. Very much the Ferrari F40 or McLaren F1 of super computing. You've seen pics, maybe even seen one at a car show, but you know you'll never be allowed to touch one. It had as much class as an Italian sports car too.
I find myself wondering how many /. geeks it would take at what $$ amount to colocate a community Cray somewhere ...... be like going in with 100 friends to buy a Ferrari though. Who gets to keep the keys?
Of interest to Cray-3 info (Score:3, Interesting)
Cray-3 memories by Steve Gombosi From a comp.unix.cray posting
Graywolf ("S5") was installed at NCAR. Like all NCAR supercomputers, until fairly recently, it was named after a Colorado locale.
This was the *only* Cray-3 shipment, installed in May 1993, the machine was a 4-processor, 128 Megaword system.
Two problems in the Cray-3 system were uncovered as a result of running NCAR's production climate codes (particularly MM5): a problem with the "D" module causing intermittent problems with parallel codes, and an error in the implementation of the square root approximation algorithm which caused incorrect results for certain data patterns (kinda like the Pentium divide bug ;-) ). These were rectified and replacement CPU modules were installed, although I can't remember the date.
The machine ran NCAR production until CCC folded in March, 1995. Since NCAR never paid for it, at some point we reduced the CPU count to 2 and let the machine run essentially unattended. I'm not too sure when that happened, although it marked the end of my regular commuting between Colorado Springs and Boulder.
There were a total of 7 Cray-3 "tanks" constructed. S1-S4 were single "octant" tanks (the smallest that could be constructed) which accomodated up to a 2 processor/128MW configuration. S5 and S6 were two-octant tanks. S7 was a four-octant tank which we used as a software development and benchmarking platform. S6 was chiefly used for system testing.
S1-S3 were diverted to Cray-4 testing once the Cray-4 project built up steam. S4 was diverted to the quite possibly suicidal Cray-3/SSS project after S7 became available (S4 was previously our software development machine).
For those of you who have Cray-3 posters lying around (by the way, I took all the photos on that poster as well as the Cray-3 and Cray-4 brochures and all the annual reports except the first two):
1) The big photo is of S5 ;-)
2) Seymour is leaning on S5 (and you have no idea how hard it was to get him to hold still that long while wearing a suit...or to talk him into that particular pose)
3) The two "cooling system" photos are S6
4) The hand holding the module is mine
Cray-3 modules were 4x4x0.25 inches in size. Each module consisted of a multi-layer "sandwich" of PC boards (69 electrical layers), with 2 layers of 16 1x1 inch stacks. The stacks were the circuit boards containing the actual circuits (GaAs for logic, SRAM for memory modules). There were 16 bare GaAs chips mounted to each side of a logic stack. I think there were 12 bare SRAM chips on each side of a memory stack (the logic chips were square, the memory chips were rectangular).