Catch up on stories from the past week (and beyond) at the Slashdot story archive


Forgot your password?

Cray XT-3 Ships 260

anzha writes "Cray's XT-3 has shipped. Using AMD's Opteron processor, it scales to a total of 30,580 CPUs. The starting price is $2 million for a 200 processor system. One of its strongest advantages over the std linux cluster is that it has an excellent interconnect built by Cray. Sandia National Labs and Oak Ridge National Labs are among the very first customers. Read more here."
This discussion has been archived. No new comments can be posted.

Cray XT-3 Ships

Comments Filter:
  • How big is it? (Score:3, Interesting)

    by rooijan ( 746599 ) on Tuesday October 26, 2004 @03:36AM (#10629031) Homepage
    I read the article (okay, so I kinda read it :-) ) and it has the speed and specs to be a geek's improvement on sliced bread. But how big is it, physically?

    The article doesn't appear to mention its dimensions, and I'm curious to know what kind of space you need to install this baby. Anyone got any idea?
  • real FPU operations (Score:5, Interesting)

    by Barbarian ( 9467 ) on Tuesday October 26, 2004 @03:38AM (#10629039)
    How are the Opterons at standard FPU operations in double precision? SSE2 and friends are nice, unless you have to make compromises in your simulations.

    I ask, because I remember that the Athlons beat the pants off the Pentium 4's in FPU operations, so all the benchmarks were rewritten to use SSE2.

  • Heat... (Score:1, Interesting)

    by Gurezaemon ( 663755 ) on Tuesday October 26, 2004 @03:38AM (#10629040)
    I can't even begin to imagine what sort of heat this thing would put out, or what sort of power requirements it would have.
  • by Henriok ( 6762 ) on Tuesday October 26, 2004 @03:48AM (#10629085)
    It seems that the XT-3 not only use Opteron processors but they also use PowerPC 440 co-processors from IBM to off load inter-processor communication from the main computing CPUs. Quite an interessting set up.

    The XT-3's biggest comptetitor in this segment must be the BlueGene/L type super computer made by IBM. The processors in Blue Gene/L is a custom built dual core version of the PowerPC 440 with built in high speed interconnects.

    Just like IBM have a finger in all the future game consoles, they seem to have a finger in several of the next generation super computers also. Nice going IBM.
  • Intersting note (Score:3, Interesting)

    by floydman ( 179924 ) <> on Tuesday October 26, 2004 @04:46AM (#10629234)
    from their Tech.sheet [] they are using the Luster file system []

    This is the first time i see a shipped linux with this file system. Now the intersting part is that lusterfs is made for linux clusters, but this monster is not a cluster... any body can shed some light?
  • Re:imagine a... (Score:2, Interesting)

    by catch23 ( 97972 ) on Tuesday October 26, 2004 @05:05AM (#10629287)
    Man a single node is $20,000?? I could build a pretty good opteron system for $2000.... so the other 18 goes to the interconnect? $2 million is a lot for only 200 processors.
  • by jmv ( 93421 ) on Tuesday October 26, 2004 @05:07AM (#10629300) Homepage
    Couple facts about SSE:
    1) You can use it in scalar mode, in which case it's almost like x87, only a bit faster because:
    a) It doesn't use a braindead register model (stack)
    b) On P4, you can do a mul and an add in parallel with SSE, but not with x87
    2) You can use SSE intrinsics. It's not as easy as "normal" programming, but easier than assembly and almost the same speed.
    3) Unaligned access is possible. It's slower than aligned access, but overall better than non-vectorized code.
    4) Trig is so slow that SSE/x87 doesn't matter (unless you write approximations, in which case SSE will also be faster).
  • Re:cray (Score:5, Interesting)

    by Anonymous Coward on Tuesday October 26, 2004 @05:12AM (#10629311)
    Cray never went "belly up". It was acquired by SGI around 1997 or so, then divested and merged with Tera, who renamed the resultant entity "Cray Research".

    Although it's true that Cray was not growing strongly before the SGI buy-out, it was not failing either. It could have kept running quite happily for many years, but in the bizarro-world of Wall Street, a company which is not growing is dying. I so love it when economists use biological terminology for corporations. In Wall Street's thinking, the only healthy growth would be a cancerous tumor.


    The whole SGI-period of Cray is actually quite fascinating, and I suspect the true story will never be fully known. Lots of SGI engineers had their non-Cray technology branded with Cray marketting names, most egregiously LegoNet becoming CrayLink. Lots of Cray folks - aka. Crayons - felt that the core of their company was gutted by an SGI operation which didn't care for the extreme high-ends of HPC.

    One rumor I heard, from a well-placed source, is that the Cray merger with SGI was primarily arranged by the USG. The intelligence services have huge investments in both company's products, so the merger between them made sense. I was told that as a quid-pro quo, the USG had an in-principle agreement to continue purchasing Cray gear to provide enough revenue inside SGI to keep both Cray architectures alive. However, certain parts of SGI felt that the US government didn't live up to their agreement, negotiations to rectify that weren't successful, and so SGI management defunded significant aspects of the Cray engineering work.

    Also, FYI, Cray is one of those companies which will never totally go "belly up" anyway. Given the sensitivity of the work which they did, their support databases alone are full of sensitive and/or classified information. Should the company cease trading, it would be acquired by a shelf company whose sole function is to ensure this data would remain private. That's been the fate of almost all of the now-defunct supercomputer and high-end graphics companies who formerly supplied the defence and intelligence market.
  • by fbform ( 723771 ) on Tuesday October 26, 2004 @05:13AM (#10629312)

    More interesting is this spec:

    Acoustical Noise Level: 75 dBa at 3.3 ft (1.0 m)

    For comparison, that's roughly the same as an average vacuum cleaner when you're operating it, or maybe a good-sized pickup truck passing you in the next lane.

    And remember, this value is *per cabinet*. You have to do a weighted sum over all the cabinets in an installation to get a true dB level. I wonder whether the maintenance people will have to use noise-level exposure limits for this baby.

    And here I was, complaining about the quiet whine of my PC's fan.

  • by pchan- ( 118053 ) on Tuesday October 26, 2004 @05:26AM (#10629331) Journal
    Power: 14.8 kVA (14.5 kW) per cabinet.

    that's amazing. how did the cray guys get a kilovolt-ampere that is not equal to a kilowatt? just goes to show you the power of fast interconnects.
  • Re:newfangled buzz. (Score:3, Interesting)

    by adzoox ( 615327 ) on Tuesday October 26, 2004 @05:36AM (#10629355) Journal
    Ha - well I'm sure the guy behind CherryOS will have a press release that it runs The Mac OS at 30 Terahertz.
  • by pchan- ( 118053 ) on Tuesday October 26, 2004 @05:42AM (#10629368) Journal
    let's see what you're missing:

    * first, sgi still makes and sells supercomputers, they are far from faded. they also own cray (or did).
    * tandem, bought by compaq, we all know what happened there.
    * hp sells a superdome once in a while. but nobody seems excited about their itanic systems.
    * sun, rotting with their out of date cpus.
    * fujitsu is doing well in the supercomputer market.
    * nec is also successful.
    * ibm, of course.

    and you mentioned motorola? you're joking, i hope.

    the largest purchasers of supercomputers in the world - national labs and the nsa, like to buy american hardware. they've always had a hand in keeping the industry afloat. notice that the big labs tend to round-robbin their supercomputer vendors so that they buy a machine from each vendor.
  • by Moraelin ( 679338 ) on Tuesday October 26, 2004 @07:48AM (#10629668) Journal
    The real problem that stands between scientists and them having lots of shiny toys is funding.

    E.g., yeah, having a 30,000 CPU super-computer to simulate your gene model on would be nice. Forking over half a billion for it, well, it's suddenly not that nice any more.

    Having one of those to simulate an electronic circuit, now that would probably rock. Again, paying half a billion for it, suddenly isn't that attractive.

    The real question isn't how nice a toy you'd like to have, it's ROI. (Unless you work for the government, and just have a budget you _have_ to blow on stuff, whether you need that stuff or not.)

    And in that context, you'd be surprised what you _can_ do with a lot less expensive toys.

    Having Cray's custom interconnects sure is impressive, but for a lot of problems they're not even needed any more. _That_ is what killed Cray.

    Most RL problems are not really the kind described as "_one_ huge indivisible data set, that you have to process in _one_ huge batch process." They're more like "we have this process with a small data set that we have to run 100,000,000 times." Most design problems or biology problems are really of that kind: run the same thing 100,000,000 times with different parameters.

    And as Seti@Home or Folding@Home proved, a helluva lot of those don't really need _any_ kind of shared memory or fancy interconnects. The real ticket is noting that instead of accelerating the batch run 200 times, you could just split it into 200 smaller batches ran on 200 single-CPU machines.

    The super-computer solution costs 2,000,000 just for the machine alone, while the 200 PCs solution costs 200,000 or so. I.e., 10 times cheaper. Better yet, the 200 PCs solution is also far cheaper to program. (Anyone can program a non-threaded batch app.) _And_ for that kind of a problem the 200 PCs solution would actually finish faster, since it has no contention issues whatsoever.

    Again, that's what really killed Cray and the super-computers. They're techologically impressive, they're a geek's wet dream, but... for 99.9% of the problems out there they're just not worth the price any more.
  • by ebooher ( 187230 ) on Tuesday October 26, 2004 @07:52AM (#10629678) Homepage Journal

    So come on, ante up. How many remember being awed at the mere sight of old Crays back in the day? Like the Cray-3? I remember the first time I saw a Cray .... thing was in an anti-static environment. To access it, one had to pass through an airlock and be "decharged" or "depolarized" etc. Basically they some how charged the air to get rid of static electricity. Then you had this system that was running *in* liquid! Take that "Oh I'm so cool cause I have a l337 haX0r water cooled CPU" overclockers

    They (Cray) were so proud of this accomplishment that the upper portion of the cabinet was some kind of plexiglass so you could see the fluid as it moved, and moved wiring and what not with it. Very surreal feeling, almost like the thing was breathing.

    And what about the Cray-1? Wasn't that a true testiment to 70's *art* and sculpture? The thing looks like some kind of freaky bus station bench with it's odd red and white panels and black base. Though, I don't know if they all looked like that, maybe you could get them in other colors?

    Ahh .... those were the days.

  • by ebooher ( 187230 ) on Tuesday October 26, 2004 @11:32AM (#10631230) Homepage Journal

    Looked around on the net, as well as a couple other /.'rs here, and someone posted a link here to a 2 and I found a pic of a 2 with the waterfall system that was mentioned by another person, and I must accept defeat within the loosened strands of my unraveling mind.

    It was indeed a Cray-2 that I remember so vividly. Nevertheless, still an extremely exotic machine. Very much the Ferrari F40 or McLaren F1 of super computing. You've seen pics, maybe even seen one at a car show, but you know you'll never be allowed to touch one. It had as much class as an Italian sports car too.

    I find myself wondering how many /. geeks it would take at what $$ amount to colocate a community Cray somewhere ...... be like going in with 100 friends to buy a Ferrari though. Who gets to keep the keys?

  • by ebooher ( 187230 ) on Tuesday October 26, 2004 @11:43AM (#10631384) Homepage Journal

    Cray-3 memories by Steve Gombosi From a comp.unix.cray posting

    Graywolf ("S5") was installed at NCAR. Like all NCAR supercomputers, until fairly recently, it was named after a Colorado locale.

    This was the *only* Cray-3 shipment, installed in May 1993, the machine was a 4-processor, 128 Megaword system.

    Two problems in the Cray-3 system were uncovered as a result of running NCAR's production climate codes (particularly MM5): a problem with the "D" module causing intermittent problems with parallel codes, and an error in the implementation of the square root approximation algorithm which caused incorrect results for certain data patterns (kinda like the Pentium divide bug ;-) ). These were rectified and replacement CPU modules were installed, although I can't remember the date.

    The machine ran NCAR production until CCC folded in March, 1995. Since NCAR never paid for it, at some point we reduced the CPU count to 2 and let the machine run essentially unattended. I'm not too sure when that happened, although it marked the end of my regular commuting between Colorado Springs and Boulder.

    There were a total of 7 Cray-3 "tanks" constructed. S1-S4 were single "octant" tanks (the smallest that could be constructed) which accomodated up to a 2 processor/128MW configuration. S5 and S6 were two-octant tanks. S7 was a four-octant tank which we used as a software development and benchmarking platform. S6 was chiefly used for system testing.

    S1-S3 were diverted to Cray-4 testing once the Cray-4 project built up steam. S4 was diverted to the quite possibly suicidal Cray-3/SSS project after S7 became available (S4 was previously our software development machine).

    For those of you who have Cray-3 posters lying around (by the way, I took all the photos on that poster as well as the Cray-3 and Cray-4 brochures and all the annual reports except the first two):

    1) The big photo is of S5
    2) Seymour is leaning on S5 (and you have no idea how hard it was to get him to hold still that long while wearing a suit...or to talk him into that particular pose)
    3) The two "cooling system" photos are S6
    4) The hand holding the module is mine ;-)

    Cray-3 modules were 4x4x0.25 inches in size. Each module consisted of a multi-layer "sandwich" of PC boards (69 electrical layers), with 2 layers of 16 1x1 inch stacks. The stacks were the circuit boards containing the actual circuits (GaAs for logic, SRAM for memory modules). There were 16 bare GaAs chips mounted to each side of a logic stack. I think there were 12 bare SRAM chips on each side of a memory stack (the logic chips were square, the memory chips were rectangular).

Executive ability is deciding quickly and getting somebody else to do the work. -- John G. Pollard