Cray XT-3 Ships 260
anzha writes "Cray's XT-3 has shipped. Using AMD's Opteron processor, it scales to a total of 30,580 CPUs. The starting price is $2 million for a 200 processor system. One of its strongest advantages over the std linux cluster is that it has an excellent interconnect built by Cray. Sandia National Labs and Oak Ridge National Labs are among the very first customers. Read more here."
Re:How big is it? (Score:4, Informative)
Weight (maximum): 1529 lbs per cabinet (694 kg)
http://www.cray.com/products/xt3/specifications
Just the name brings back memories (Score:3, Informative)
The name is synonymous with speed and power and the unwillingness to cut corners in order to shave a few dollars off the final product. When you buy a Cray, you know you are getting top of the line hardware.
It looks like Sandia wants to build the fastest supercomputer in the world by clustering a few of these monsters, and I have no doubt that they will. Looks like more fun articles about this in the future.
There are two prominent applications for these machines. The first is nuclear weapons simulation. Personally, I don't see the point to that. The other application is in weather prediction. By feeding in current weather variables into a well-written model, a supercomputer is able to predict to a large degree of accuracy the future weather. Such an application will always be welcome.
I think I'm going to have to fire up the old ][e, the nostalgia is killing me!
Re:real FPU operations (Score:5, Informative)
How big it is (Score:2, Informative)
Dimensions (cabinet):
H 80.50 in. (2045 mm) x W 22.50 in. (572 mm) x D 56.75 in. (1441 mm)
Sorry to reply twice but I forgot this detail.
You don't have to begin to imagine (Score:4, Informative)
Re:So......the cost compared to? (Score:2, Informative)
Re:Opterons and PowerPC together (Score:3, Informative)
Re:software (Score:5, Informative)
UNICOS is usually a safe bet. In this case the specs [cray.com] say UNICOS/lc, which is made up of "SUSE(TM) Linux(TM), Cray Catamount Microkernel, CRMS and SMW software"
I'm not entirely clear how to interpet that, but I think it runs as follows: It runs the Catamount Microkernel as the kernel, and uses SUSE for everything else (so we have SUSE Linux, without the Linux - all of a sudden that GNU/Linux stuff starts to make sense). The CRMS is their interconnect management and monitoring software, and SMW is the System Management Workstation - which I'm guessing is their administration frontend.
It's worth noting that that's some pretty serious software there (because Cray has a lot of experience dealing with large systems) - you can bet that the management and monitoring software is some very serious stuff.
This thing is to a beowulf cluster what a dual G5 PowerMac is to homebuilt PC system running Linux From Scratch. It's going to work flawlessly "out of the box" with a smooth and polished interface that lets you get done everything you want to do simply and easily. You can of course make your home built PC with LFS work just as well, it's just going to take you an awful lot of effort.
Jedidiah.
Re:So......the cost compared to? (Score:5, Informative)
There's not a lot to compare. We're talking apples and oranges. It's like asking to compare a PowerMac G5 with a bunch of PC parts scattered on the floor as desktop machines. Sure, you can put the PC together, load it with Linux, tinker with it to get everything working, etc. but that's a fair amount of work compared taking the PowerMac out of the box, plugging it in, turning it on, and having everything work perfectly.
Read the specs [cray.com], particularly with regard to the interconnect, system administration, and hardware and software reliability features. This thing is seriously engineered to be massively parallel system with top of the line hardware and software to support and maintain that, as well as extremely impressive reliability features.
Jedidiah.
Re:MP performance overhead (Score:1, Informative)
The algorithm will decompose the workspace so as each node can compute its own slice of the problem almost independently.
The is only a slight communicatiosn overhead depending on what application is run. This is also offset a bit by the supposedly fast interconnect hardware.
Re:MP performance overhead (Score:4, Informative)
You can't really compare something that can hold thousands of CPUs to something powered by Abit that can hold two, anyway. It's like comparing apples and a strange bug thing with tentacles.
Re:You don't have to begin to imagine (Score:1, Informative)
14.5kW would just be the real component of power, i.e. heat dissipated
this leaves - using pythagoras' theorem - an imaginary component of power of 2.96kVA
Re:You don't have to begin to imagine (Score:5, Informative)
It's fairly common to get a KVA !=KW.
Overall power used by a load is expressed as S=P+jQ, where P is the "real" power and Q is the reactive power (capacitive/inductive from motors, fluorescent lamp ballasts, etc).
While the "units" of S, P, and Q are power=voltage*current, S is generally expressed in VA, P in W, and Q in VAR(volt-ampere reactive) to differentiate the variables. Because the magnitude of S=sqrt(P^2+Q^2), S will always be greater than or equal to P (in this case, 14.8kVA=sqrt((14.5kW)^2+(+-2.965kVAR)^2)
Re:Just the name brings back memories (Score:5, Informative)
There are two prominent applications for these machines. The first is nuclear weapons simulation. Personally, I don't see the point to that. The other application is in weather prediction.
Oh, please. Buy a clue, will ya? There's lots and lots and lots of applications that use supercomputers, or could use if they were more affordable. A few examples from the top of my head:
Materials science, that is ab initio simulations, moldyn, you name it. This alone probably uses > 50 % of all supercomputer cpu time in the world. By comparison, weather prediction and nuke simulations is small potatoes (or shall we say, the simulations as such are big, but the number of people engaged in weather prediction or nuke simulation is really small compared to all the supercomputing materials scientists).
CFD, the automobile and aerospace sectors are big users.
Electronic design.
Seismic surveys, the oil industry uses lots and lots of supercomputers to find oil deposits.
Biology. Gene sequencing, moldyn simulations of lipid layers and whatever.
Climate prediction, somewhat related to weather prediction. Official purpose of the Earth Simulator.
All of the examples above could easily use almost any amount of cpu power you can throw at them. The only thing that stands between a lot of scientists and improved understanding of the world is computing power.
Re:So......the cost compared to? (Score:3, Informative)
The VT Supercomputer specs vs the Cray specs page you pointed to:
CRAY 460 GFLOPS per cabinet (96 processors @ 2.4 GHz)
Apple - if my math is right - 420 GFLOPS (100 processors @ 2.0Ghz)
The new specs for the specialized VT Supercluster are pretty impressive.
Their throughput and interconnect is most likely weaker - but still VERY strong with fiber channel.
Re:imagine a... (Score:5, Informative)
The upper bound on speedup is generally Amdahl's law [wikipedia.org]. Plainly, the efficiency approaches zero as the number of processes is increased. Generally we consider the major sources of overhead to be communication, idle time, and extra computation. Interprocess communication is considered negligible for serial programs in this context (we consider message passing). Idle time ends up contributing to overhead, because processes idle awaiting information from others. Extra computation is virtually unavoidable at some point; for instance in MPI's Single Program Multiple Data model, each process in tree-structured communication other than the root is eventually idled prior to the completion of computation, and each process determines IPC at some point based on rank.
There are notable exceptions to Amdahl's law, however; Gustafson, Montry and Benner wrote about such in Development of parallel methods for a 1024-processor hypercube, SIAM Journal on Scientific and Statistical Computing 9(4):609-638, 1988.
Re:Sic transit gloria mundis (Score:2, Informative)
That's why they use mere 100-series Opterons: they need only one HT link per CPU. Because the whole is not based on HT interconnects.
Really, loosely-coupled cluster my ass. This machine *is* capable of record-breaking single-task performance. Read the product pages again.
hybrid system with multiple kernels (Score:3, Informative)
Re:imagine a... (Score:2, Informative)
1 CPU @ 1.00x -> 1.00 / 1 = 1.000
2 CPUs @ 1.95x -> 1.95 / 2 = 0.975
4 CPUs @ 3.20x -> 3.20 / 4 = 0.800
64 CPUs @ 50.0x -> 50.0 / 64 = 0.783
Pop that into an OpenOffice.org spreadsheet and look at the graph.
That is not linear, in fact, it's non-linear in the direction that *helps* more and more processors. If the decline from 4 CPUs to 64 CPUs is a mere 1.7% efficiency compared to the 17.5% drop from 2 to 4, then, by golly, I'm going to cram hundreds of CPUs in there and see it tail off. Hello amazing performance.
Instead, reality is that the dynamics change. You can't evaluate "equivalent performance" to a single processor system. There is no reasonable metric with which to do so.
-Ant Slayer-
are you sure you remember seeing the Cray 3 ? (Score:3, Informative)
Re:... Back in my day .... young whippersnapper (Score:2, Informative)
Before that was the Cray-2 (a.k.a World's most expensive aquarium")? In case anybody's interested, I believe they used Fluorinert as the liquid, as it wouldn't swell the PC boards, short anything out, or cause anything to corrode.
A note, the Cray-3 was created by Cray Computer Corporation of Colorado, whereas the Cray-1 was made by Cray Research of Wisconsin. In ~1990, Seymore wanted to start working on computers using gallium arsenide instead of silicon, since they could switch faster. Cray Research didn't want to try anything so revolutionary, so Seymore headed to Colorado with a group of people and started CCC. Unfortunately, they apparently made exactly one Cray-3 [cray-cyber.org], then folded.
Seymore Cray was quite the Übergeek.
Re:The math for a comparable Xserve system (Score:3, Informative)
#1 RAM: $3000 for the G5 cluster node includes 512mb ram. Most places demand atleast 2gb ram per CPU, we require 3GB ram per CPU in all new system purchases. This brings the node price (from apple.com) to $6500
200x $6500 = $1,300,000
#2 Racks and power: Each rack can hold about 32 machines (without getting way to hot/dense) for 200 nodes, this would be about 7 racks.
7x $1200 = $8400
#3 Interconnect: No HPC system is usefull without an interconnect. An 80 node myrinet system was $250,000, so at $3125/node you're looking at:
200x $3000 (estimate) = $600,000
#4 Networking: you need a network switch and cabling to connect all the nodes... gige is a must these days. Let's say we go cheap with HP ProCurve 2848 Layer2 managed for $3300 each we need 7 of those, one for each rack cabinet.. with trunking we can get 4gb back to a central switch. not too bad. Say we add $10/cable for pre-made patch cables, (length averaged) that's about $2250 in cables.
7x $3300 + $2200 = $25,300
#5 Disk: You quoted a bunch of XservRaid's without any kind of apple care.. with IDE raid.. I'm not going without some kind of support on it. Oh wait.. 1 file server is NOT enough to handle 200 nodes of HPC.. and apple doesn't have a clustered filesystem. You're going to have to go with Linux/Intel with RedHat GFS for that one (yes, there are other options, but I know GFS)
Say we do 4 XserveRaid's with applecare:
4x $16,000 = $64,000
We also need for dual whatever intel machines: (i'll be nice and include F-C cards in the price)
4x $3000 = $12,000
We also need a F-C switch to link all the nodes:
SanBox 8 port $5200 and 8x SFP modules $750 = $11,200
I'll pretend like we don't need GFS software support, but most places would want it. (it's another $20,000 or so, but eh.. we want cheap solution)
Disk total comes to: $87,200
Price so far: $2,020,900
And that doesn't even include setup!
Re:software (Score:4, Informative)
catamount is the kernel that runs on the compute nodes. IT's a tiny kernel that packages up the OS service requests, and sends them, over the interconnect, to an OS or I/O node, which does the real work of the operating system. catamount is a descendant of PUMA, which came from Cougar. These are heavily derived from work done at caltech. (I believe CMU, and one of the UTexas schools also played a role, but am not sure). The idea is that the microkernel is small and unobtrusive, and it gets the hell out of the way so the application can use the CPU as much as is possible.
The OS and I/O nodes run linux, and provide services to the compute nodes. This is probably, but it could just as easily be running as a user-space daemon on the OS node. (Though you might have to do some mem-copys that way, which would lower performance)
NOTE: Though these nodes take advantage of some of linux's features (like the lustre file system) they do NOT necessarily implement these features for the system as a whole. They probably provide a minimal set of features necessary for the sorts of problems that the xt3 runs. All the scheduling work that has gone into more recent linux kernels is of little use, as the compute nodes have their own scheduler, probably more closely tied to the batch dispatcher than to the linux kernel. To say that the system runs linux is true, but a little misleading. It's a very different linux than what runs on my desktop, and it's used in a very different way.
Wall Street (Score:3, Informative)
Surrealistic point in case: at one point 3Com had a lower market value than the Palm daughter-company. Basically if you subtract the value of the Palm shares, the whole rest of 3Com was actually worth a _negative_ value for the stock market.
And we're talking divisions which were making a tidy profit. Yet they were apparently worth a _negative_ number.
No, it's not a joke. Roll it around a bit in your head to fully grasp how completely sad and idiotic that is. Real profits, real assets, worth a negative number of dollars. Stupid.
Or at the other end of the spectrum you have Microsoft whose stock market value is _way_ above the value of its assets. Without paying any dividends or acquiring much in the way of long term assets, people just flocked to drive the price up and make Bill Gates rich. Basically to give their money to Bill Gates and not even get a Windows CD in return.
The thing is, however, the stock market value has _nothing_ to do with a company's value or profits. The value of a share is only worth as much or as little as people want to believe it is. It is like Monopoly (the board game, not MS;) money: if tomorrow we decide that the blue bills are worth 10% more and the red bills are worth 10% less, who's to argue with that.
The _only_ reason the stock market on the whole goes up is basically because yearly people dump more money into it. Basically it goes up just because people want to believe it's going up, and put their money where their belief is.
And the way those values fluctuate, now that just has to do with hype and greed.
The stocks worth buying are those who'll make you a profit: typically meaning they'll raise in value. The stocks worth selling are those who don't.
Except with no intrinsic value it becomes a game of guessing what the other lemmings will buy (driving the price up), and what the other lemmings will sell (driving the price down.)
One thing that makes lemmings buy is the prospect of growth. Hence, hype is good. Hence, yes, shares in a cancerous tumor would sell like hot cakes and rocket sky high in price.
Hence, conversely, shares in a company which doesn't grow or otherwise cause more lemmings to buy, are not worth holding on to. Because they won't bring a profit. If Microsoft truly plateaued and didn't pay dividends either, regardless of how much profits it made at that point, its shares would plummet. Because between holding onto a share in MS that doesn't bring a profit, and investing in some startup that grows quickly, the second promises more of a ROI.
Now that's all a bit of an over-simplification.
Of course, there are other factors. Like just paying dividends to give people a reason to hold onto your shares even without massive hype and growth. (See why MS started doing that when its market explosion slowed down.) Or like fraud: "analysts" just telling lemmings what to buy, and thus drive up the price of the shared owned by the "analyst" and his/her clients. Etc.
But as a quick intro to the madness of the stock market, it will have to do.