I'm not really a Ubuntu fan, but with the cluster I manage (120 physical cores, 960GB RAM) we've ended up going with Ubuntu 10.04 running Sungrid Engine for a couple of reasons.
- The LTS support, by the time the support period ends we should be replacing the hardware any way.
- It provides a Grid engine package by default (might not be the latest but it's good enough) for distributing the workloads
- A lot of people are already familiar with Ubuntu
- Most third party apps provide support for it
- It's very stable
- It's free
Note if your users are heavy R users have a look at installing the Revolution R package from the third party repositories. It can provide some massive speed ups for Matrix work and a number of other jobs.