Re:Dumb (Score 1)

Skip the raised floor and other stuff that isn't critical, shop carefully and buy the battery system used and you can put it together for well under $100k.

The cost of the additional building power transformers and switchgear *alone* would cost well over $100k, for the amount of power necessary. Add to that at least 85 tons of air conditioning necessary to cool the cluster (assuming 100% effiency, and no redundancy). At about $40k for each 30ton Liebert chilled/potable water DX unit, that's another $120k...

Also, if you think that we can afford to put either a UPS or generator backup under this whole mess, you don't seem to understand what kind of minimal funding we get for facilities and infrastructure. Truthfully, considering the amount of money that it'd cost vs the cost of the downtime (and typical amount/length of power outages), putting a UPS and genset under all of the cluster nodes probably isn't worthwhile, or else our customers would actually pay us to do it.

With a fiber ring, they could have (temporarily) distributed the cabinets around the campus, bringing the machine up to full power. Then once the researchers sign off on it, the old one is powered down and moved out.

Each pair of racks has a 50Gb connection back to the switching backbone. Building the cluster in this manner would horribly hurt performance (latency), probably require an extra $200k or so in networking hardware that would be useless after we re-assembled the cluster in one location, inconvenience our users (user jobs can run for up to 30 *days* at a time), require us to somehow convince other departments on campus to give us power and cooling to house the racks, require many, MANY more man hours to assemble and then move the machines back across campus, probably resulting in more hardware failures due to the number of times they're moved, etc.

It's simply not a feasible idea to try to assemble the new cluster before the old ones were demolished.

Each of the 800 machines costs at least $5000, so for the price of 20 of the machines you can build a whole new room to house them.

Each machine cost us no more than about $2300 (excluding the infrastructure). In total, we purchased about 1000 machines for about $2.6M including racks, cables, network switches, etc, so about $2600/machine total cost.

Also, for many reasons you can't just spend money on whatever you want. The money's source dictates whether it can be spent on computers, staff salaries, building improvements, etc. It's not like "here's a check for a lot of money, do with it as you please."