Cluster Interconnect Review

deadline writes to tell us that Cluster Monkeys has an interesting review of cluster interconnects. From the article: "An often asked question from both 'clusters newbies' and experienced cluster users is, 'what kind of interconnects are available?' The question is important for two reasons. First, the price of interconnects can range from as little as $32 per node to as much as $3,500 per node, yet the choice of an interconnect can have a huge impact on the performance of the codes and the scalability of the codes. And second, many users are not aware of all the possibilities. People new to clusters may not know of the interconnection options and, sometimes, experienced people choose an interconnect and become fixated on it, ignoring all of the alternatives. The interconnect is an important choice and ultimately the choice depends upon on your code, requirements, and budget."

Re:GigE FTW

[multimediavt]

You're talking bandwidth in a read/write to a filesystem. You are not taking into consideration applications that are latency bound, or are both latency and bandwidth bound when passing information from node to node, let alone writing to a filesystem. We run a number of scientific codes on our IB-based cluster. Some of these codes are slinging around up to 20GB of data passing messages between nodes, and this is memory copies not filesystem read/wries. It has to be fast (lower the latency the better for these particular codes), and it has to have a data path capable of having large amounts of data (above 512 MB/s) going in each direction (TX and RX) at the same time.

It ain't bananas, it's NUTS! But, it does happen. Of course the one application could be running on up to 900 processors (450 nodes) at a time and will generate data files to our storage system in the neighborhood of 250 GB when it's all done. YIKES!