If you look at the history of infiniband, it was always intended to be something like numalink+xio. Origionally you were supposed to connect peripherals, storage, and processor nodes onto this big network and add and remove them all dynamically. It got scaled down from that, and now is pretty much used as a high-speed network, with the occasional RAID attached directly to it. Numalink can be used in this way too. One does not need to make a single numa domain from an altix.

The numalink chip also has the extended cache directory logic in it, which allows large numa machines. Importantly, that version of large is large on the scale of numa database servers, but rather small on the scale of supercomputers. Even SGI has to fall back to infiniband for the really large machines, such as the two big systems at nasa. It's not as feature-rich as numalink, but it'll scale to tens of thousands of nodes, sorta affordably. I should note that there's no reason that the cache director chips can't talk to one another over an infiniband network. Noone has invented this chip, but the network can be an independant piece.

I agree that SGI has long had great technology, and useful products. (I reserve the term great products, as they have tended to have great strengths coupled with great weaknesses) But I would not say that their products have been successful. If they had, SGI wouldn't have been circling the bowl for the last ten years. SGI learned how to make a lot of money when they were at the top of a growing market. They never learned how to make money in a shrinking market, or how to transition to a profitable spot in a different market.

actually, I know you are mistaken about lustre. Lustre is a regular kernel filesystem just like CXFS, stornext, GFS, or GPFS. In the case of puma or bluegene you have to link it into the application, but not on linux. The point, however, remains. SGI has used CXFS to sell its hardware, which was awesome at the time, but it limits the ability of rackable to make a business out of selling CXFS as a stand-alone product. IDeally you would want to sell CXFS licenses on every commodity cluster out there, your own, or the competition. Sun has that now, with lustre. Lustre is run on IBM clusters, HP clusters, even SGI clusters. I doubt rackable could turn CXFS into that product and displace lustre from very many machines.

SGI has long suffered from the classic theme of the innovator's dilema. They invented really cool graphics technology in the 80s and early 90s. It performed very well, but they charged an enourmous amount of money for it. Along comes the first generation of 3D graphics cards for PCs. At that point, SGI had the option of putting out a top notch PC graphics card. They could have become a dominant player in that market. Some business unit at SGI would sit in the place now occupied by nvidia. That's hard to do, as the graphics card division would undercut sales from the workstation division. However it's better to be undercut by an internal competitor than an external one.

Many of the old cray patents are expired, and some have gone to the new Cray Inc.

The problem on the IP front is that SGI already leveraged most of their IP a long time ago. Most of it is already sold, cross-licensed, or expired.

This is why SGI finally fell apart; you guys are all talking about SGI workstations. SGI hasn't been in the workstation business for years. There hasn't been a workstation business for years. HP,IBM,Sun sell workstations, but they are just rebranded PCs. Dec,DG,EnS,Intergraph,Appalo: all defunct.

Lately SGI has been selling low-end HPC clusters and a few mid-range altix machines. (and one really big one at nasa) The HPC business is a really difficult place to make money. SGI has never been good at keeping their operating costs down. Compared to their competition, they always seemed to employ a lot of people, and have a lot of irons in the fire, most of which never panned out.

SGI has always loved to engineer their way around problems; In a mature market one makes money by engineering a solution to a problem and then licensing it out to the rest of the world until it becomes an industry standard. Numalink could have been what infiniband is now. Infinitereality could have been what geforce is now. CXFS could have been what lustre is. XIO could be PCIe. SGI wanted to control it though. They tried to keep it all under the tent.

I'm sure they're not picking up the debt. Rackable doesn't have the assets to pick up that debt. They are picking up the company for essentially nothing, but SGI has lost money every quarter for years. So they can expect to take on those loses for at least a couple of quarters. They won't owe the creditors, but they still have to pay some sort of severence to all the people they let go, and figure out how to do something with SGI's customer list and try to turn it into new rackable business.

There are valuable parts to SGI. The CXFS/DMF business is valuable all by itself. Numalink is a good technology, but rackable isn't really in a possition to productize it in a profitable way; I'm not sure who they could sell/license it too either. Apart from that, they are buying the customer list, the sales team, and a few government contracts.

I'd love to see each core on a massively multicore design get its own memory controller. I'm not holding my breath, however. If you think of a 32-core CPU, it's pretty unlikely that most supercomputer or cluster vendors are going to pay for 32 dimms for each cpu socket. So then you're talking about multiple memory channels per memory stick. You can still get ECC using 5 memory chips per channel, so you can imagine 4 channels fitting on a memory riser. Cray does this on the X2. Then 32 channels would only require 8 dimms, which is reasonable. Then what do you do for 64-core CPUs?

It's tricky, and the problem for the market is that it's expensive. Can you get the commodity CPU vendors interested in such a thing, given that most of their addressable market is not in the supercomputing space?

I think We're gonna see more cores in a CPU that there's bandwidth to use. They might increase the bandwidth a bit, but probably just enough to get good linpack numbers.

vector processing in commodity designs isn't enough. Of course we are going to see it, at this point it's not very expensive to add. Adding vector processing for increased flops is easy. The hard part is the bandwidth. One of the reasons the X1 processors were expensive, was that they were custom, but so are the network chips in commodity-cpu supers, and they only add $1000/node. The real cost of X1-style memory is that you have 64 channels of memory, which is a lot of wires, dimms, memory parts, etc. There's a very real cost to all the memory components needed to get the kind of bandwidth you need to support a high-throughput vector pipeline.

The commodity processor vendors aren't going to do this sort of thing, as it adds to the cost of the chip, but provides nothing to the bulk of their customers who are running mysql, apache, or halflife.

The one hope I have is something like the core2 architecture, where ddr3 is used for desktop processors, and fbdimm is used for server parts. The two components share a lot of architecture, and only a few of the asic cells are different. If a cpu vendor were interested in the HPC market, they could design a cpu to use a standard memory channel for desktop/low-end server parts, and something more expensive, but higher bandwidth for the HPC space. It would mean HPC specific processors, but sharing most of the engineering with the commodity part. Maybe Cray could license them the design for their weaver memory controller in the X2. It's kind of like the AMB on a FB-DIMM, but it includes 4 channels of DDR2 on each stick of memory.

Not quite.
see http://www.pnl.gov/science/images/highlights/computing/cray.pdf
http://www.pnl.gov/topstory.asp?id=320
http://www.cacr.caltech.edu/news/story.cfm?ID=30

The problem is that no idea doubles the rate at which supercomputers advance. Most of the ideas out there jump foreward, but they do it once. Vectors, streams, reconfigurable computing. All of these buzzwords once were the next big thing in supercomputing. Today everyone is talking about GPGPUs. None of them go very far. How much engineering goes into the systems? How long does it take to get to market? How difficult is it to rewrite all the algorithms to take advantage of the new machine? What proportion of the codes see a real advantage on the new machine? Can your company stay afloat long enough to reap the rewards? (remember that supercomputing is a tiny niche market compared to computing in general.)

I've seen a lot of "game changing ideas" come along in the supercomputing world. Commodity computing is the only one left.

Back in the 90s, there were custom super-computer processors (both vector and scalar), that were faster than desktop processors for all supercomputing tasks. This hit a wall, as the desktop processors became faster than the custom processors, at least for some tasks. If you can get a processor that's faster for some tasks and slower for others, but costs 1/10th the price of the other, you're probably going to go with the cheap one. The world has petaflop computers because of the move to commodity parts. Noone could afford to build 160,000 processor systems from YMP processors.

btw, multi-cores are pretty terrible for desktop applications. They really excel for server transaction processing, but most desktop users haven't any use for more than 2 cores. A radical shift in programing is going to be needed before massively multi-core processors are any use to a desktop user.

NEc still makes the SX9 vector system, and cray still sells X2 blades that can be installed into their xt5 super. So vector processors are available, they just aren't very popular, mostly due to cost/flop.

A vector processor implements an instruction set that is slightly better than a scalar processor at doing math, considerably worse than a scalar processor at branch-heavy code, but orders of magnitude better in terms of memory bandwidth. The X2, for example, has 4 25gflop cores per node, which share 64 channels of DDR2 memory. Compare that to the newest xeons where 6 12 gflop processors share 3 channels of DDR3 memory. While the vector instruction set is well suited to using this memory bandwidth, a massively multi-core scalar processor could also make use of a 64-channel memory controller.

The problem is about money. These multicore processors are coming from the server industry. web-hosting, database-serving, and middleware crunching jobs tend to be very cache-friendly. Occasionally they benefit from more bandwidth to real memory, but usually they just want a larger L3 cache. Cache is much less useful to supercomputing tasks, which have really large data-sets. The server-processor makers aren't going to add a 64-channel memory controller to server processors; it wouldn't do any good for their primary market, and it would cost a lot.

Of course, you could just buy real vector processors, right? Not exactly. Many supercomputing tasks work acceptably on quad-core processors with 2 memory channels. It's not ideal, but they get along. This has put a lot of negative market pressure on the vector machines, and they are dying away again. It's not clear if cray will make a successor to the X2, and NEC has priced itself into a tiny niche market in weather forcasting, that is unapproachable by other supercomputer users, for price reasons.

right you are. The contracts for these machines were signed a couple of years ago. They might have sped things up, in order to get on the top500 list, but they didn't add hardware just for a little showmanship. These labs can afford to put out a bunch of press releases related to top500, but they don't care enough about it to spend many millions of dollars.

The list reflects the computers, the computers don't exist for the sake of the list.

I think it's amazing power workstations lasted as long as they did. SGI quit the biz years ago, DEC is lost in the annuls of history, sun sells workstations, but they are just PCs. The workstation market is gone, history, no more. It's been almost 10 years that an intel/linux box with a good gamer graphics card has been outrunning dedicated workstations costing ten times as much.

The press releases from Oak Ridge and Cray (unlike the summary on slashdot) were careful to claim jaguar as the fastest computer for "open science". They were, no doubt, aware that Los Alamos might have bought more hardware since June.

The machine at Los Alamos is used for classified Department of Energy projects; probably simulating nuclear warhead functionality on the aging pile of B83s and B61s sitting in the US arsenal.

The machine at Oak Ridge gets used for unclassified research that ends up in peer-review journals, on a variety of topics. Stuff like Climate models, Fusion energy research, protein synthesis models, cosmology. I'm sure there's a little competiton between the labs, but they really have different missions, so I bet they don't put too much stock in it.