Best Use For A New SuperComputer (HPC)

Supp0rtLinux writes: In about 2 weeks time I will be receiving everything necessary to build out the largest x86_64-based supercomputer on the east coast of the US (at least until someone takes the title away from us). Its spec'd to start with 1200 servers with dual socket, six core configs. We primarily do life-science/health/bio related tasks on our existing (and fairly small) HPC. We intend to continue this usage, but to also open it up for new uses (energy comes to mind). Additionally, we'd like to lease out access to recoup some of our costs. So what's the best Linux distro for something of this size and scale? Any that include a chargeback option/module built-in? Additionally, due to cost, we have to choose either IB or 10GbE for the backend, we cannot have both. Either way, all nodes will have 4 x 1Gbps ports available. Would Slashdot readers go with IB or 10GbE if they had to choose? And last, all nodes include only a basic onboard GPU. We intend to put powerful GPU's onto the PCI-e slot and open up the new HPC for GPU related crunching. Any suggestions on the most power, Linux-driver friendly, PCI-e based GPU available?