As I already said, no we will not be running X on all the nodes. One of them will run it, with a few cores reserved for the purpose (this is the way we do it today and we have no significant issues with the arrangement). But like I say, we may very well decide to use one of the boxes from the old cluster for this task, seems like a good idea. But I still think we'd like the same distro to make the administration bit easier.
One of our most used softwares is practically closed source (to get access to the source you have to pass a series of security screenings as well as motivate why you really REALLY need the source, it's almost impossible for non-US citizens). This software does not implement GPGPU functionality at this point, and whether or when it will be supported is unknown to us. Therefore, getting GPUs at this point would probably not be the best idea. It has been discussed, and we might look at getting boxes that can be upgraded with GPUs in the future.
As for being an administrator, I volunteered for it so I'm not "unsuspecting" :) I have my reasons for wanting to do it, won't go into them here. But it will not count towards the time I have for research, that is already reserved specifically for research. We will be at least 3 people sharing the load on this too, I don't really see a problem with it.
Yes, we have looked at the other clusters around here. The problem with them is that we have to pay for them per unit of processing time per specific project. These will be used for larger jobs that our local cluster cannot handle, the local one will mainly be used for developing software to run on the larger ones as well as running smaller simulations and verifying algorithms. This is very useful to us, since it would be a pain to plan small things like that ahead of time. And we do need more processing power than we have today, even smaller jobs of the type we do require some power.
Our current setup works fine for this and it has been of great use, but it is getting old and is beginning to be a limitation. So now that we're considering upgrading we wanted to do some research first, hence the question.
Thanks very much for your advice, I will keep your comments in mind. Maybe we can find someone else who wants to run a cluster for a similar purpose. Though we really do like having one reserved just for us as well. This is a general opinion in the department, and it would probably be hard to convince everyone to do it any other way. Any additional comments are of course welcome!