Comment HPC Planning (Score 1) 3
You're about to receive a large amount of hardware from the vendor, and you haven't decided upon which GPU's to use, which interconnect for communications, what OS would be appropriate, or the types of workloads your users will be running (beyond your base set)? Really? If that's the case, no amount information from slashdot will solve your problems.
If you have no interconnect chosen, how will you rack the systems in the case that cable lengths are an issue, as they are for IB? Do you even have nodes that natively support both 10GBe and IB? I highly doubt it. What about you core network switches- 1200 ports (plus switch fabric) of IB or 10GBe might cost more than those 1200 nodes. You're also talking about adding GPU's and a high speed network adapter to each of 1200 nodes; what kind of manpower do you have for the task of installing 2 PCIe cards per node for 1200 nodes. I'm assuming that you'd want to be in operation sometime before Christmas. I won't even ask about what kind of large scale storage you have planned. I shudder to think of what power and cooling requirements you've already overlooked or made impossible.
Who's your vendor? If they really let you purchase 1200 nodes without any sort of planning, they should be dragged behind horses and shot. What a waste of money.
I'm sorry to be so negative, but you guys really screwed the pooch on this one. When you are designing a supercomputer, the very first thing to be decided is what the use cases are, especially if you're trying to generate revenue from the system. You have a limited amount of money to buy computing power, interconnect, storage, and facilities, so you have to optimize your purchase in those areas around to the expected use of the system. Not to mention operating costs.
Sheesh. I hope you're just pranking us.