Comment Use IB, CentOS or SciLinux, and xCAT (xcat.org) (Score 2) 387
IB is faster and cheaper than 10GE. Unless you get 10GE from your IB vendor.
All IB solutions support RH distros fairly well, so I'd stick with RH-like or RH-proper. CentOS has been our x86 Linux reference platform for xCAT development.
Use xCAT for cluster management and use xCAT's stateless provisioning (no need for local HDs). With xCAT we were able to provision the fastest system in Canada (~4000 nodes) over 40:1 blocking GigE in 8 minutes (but we had 10 10GE-based service nodes). xCAT was also used for the first 1.0 and 1.1 Petaflop system (LANL Roadrunner).
For billing and chargeback consider Moab with Gold. If you use Moab with xCAT and stateless provisioning, then you can power up nodes on demand and power them down automatically when not in use and track/bill one energy usage. You also have the ability to specify different OS loads on-demand so that your system can be more of an HPC cloud and not just a static homogeneous cluster. Lastly xCAT can support KVM if you want to throw a few VMs in there as well. Oh, and if get the itch to use Windows, xCAT supports that too.