What Cluster Management Software is Out There?

Dreddlox writes "I'm looking at a company producing cluster management software for Linux-based clusters such as Beowulf and would like to get a feel for waht's out there right now. I know Turbolinux for example has a product, enFuzion as do a few others. I'm trying to get a complete list of the players in this market as well as on any open source software and distributions of it if any exist. Also if anybody has any pointers or information on the use of Linux clusters in non-military and non-academic applications (such as auto manufacture and finance) especially outisde the US. I'm curious as to potential for such software in Europe and Asia (with all the talk of China being a ripe market for Linux due to the open source angle)."

Cluster Management Software

[Anonymous Coward]

Look at MOSIX [mosix.com]

more cluster management software

[Anonymous Coward]

you can also look at http://www.epm.ornl.gov/torc The c3, m3c, and OSCAR pages may be of interest. M3C and OSCAR are not finished yet, c3 should be done (version 2.5, not 1.0). This page covers the work done by a group at Oak Ridge National Laboratory. [e-mail]

VA Linux

[Garfunkel]

Checkout vacuum (sp?) from VA Linux. I beleive it is an open source Cluster Management package.

Re:Possibility of China as a Linux Market

[titus-g]

Actually the Chinese are developing their own distro, Red Flag Linux (possibly more), there already has been quite a lot of work done to support Chinese language under linux even before that.

The Government there isn't really too chuffed with windows as the chinese language version was done in Taiwan, and the coders bunged in loads of anti PRC easter eggs.

I think in general the people there do still tend to use windows, for the same reason most people in userland do, shallower learning curve, more apps.

Having said that if Red Flag gets going then there could be a lot of conversions, as it is likely to handle Chinese far more natively than windows.

And finally what I really wanted to say...

Cluster Management? Try Preparation H

Re:Possibility of China as a Linux Market

[titus-g]

oh for an edit facility.

P.S. Can't see them being all that happy about the NSA backdoor in windows rumours either.

Beowulf management.

[VWswing]

This is a ripe subject. First off, go to the beowulf mailing list. After that your best place to go is http://www.beowulf-underground.org That's a slashdot-like site devoted to beowulf.. Everything is there.

Also, http://www.xtreme-machines.com/x-links.html has a lot of links to documentation. Also Bulding Linux Clusters by David Spector has a good list of software in the back.

Re:Flight of the Condor

[scotfree]

um, call me paranoid, but I thought I should clarify that there are many teams within the Genome Project, and I'm only addressing my personal experience with one such.

There is no uniform approach to this throughout the various labs, and even someone in the same group as me might give a completely different report. And it might well be better informed...:) I'm a user, not an admin of our cluster.

PBS

[fillekes]

At SailFlow [sailflow.com], we use a combination of pbs and rsync. This is because the granularity of the parallelism in our CFD apps is currently rather large -- we just queue up jobs, and they run faster if each one gets its own CPU, that's all. Works for me, and is appropriate to our app. But, in reality...there is no "better" or "best" there's just what's appropriate to your application, and whether there's a stable version that can actually do the job. It's not rocket science.

Possibility of China as a Linux Market

[Kierthos]

Actually, this is fairly intriguing. China has a reputation for software piracy, but the open source quality of Linux makes one of the more attractive OS's in the terms of that using it does not provoke reactions from Microsoft in terms of stealing the software.

Of course there is also the issue of translating Linux into Chinese, but I figure that is trivial based on the amount of software that is available in various languages. (Although seeing how certain commands are translated into Chinese may make for some humourous reading.)

Just a thought...

Kierthos

Re:Possibility of China as a Linux Market

[Kierthos]

I'm not saying that China doesn't have the right. What I'm saying is that China has a long history of commerical pirating of software, hardware, music, etc.

Because of this history, they have nothing to lose and everything to gain by using Linux. Since it is open source, it is preferable to other OS's in that they can more easily get access without hitting the 'copyright wall', whereas if they use Windows (or other OS's that are not open source) then they either have to pay for them, or as history has shown, they will pirate them. And since there is no point to pirate open source, well, you get the idea...

Kierthos

Clustering software or management software?

[booch]

If you are looking for software to create a cluster, there are several, depending upong what type of cluster you are trying to create. If you are creating a service-based cluster, check out TurboLinux Cluster Server [turbolinux.com], Linux Virtual Servers [linuxvirtualserver.org], PolyServe Understudy [polyserve.com], and Legato [legato.com]. There are many others available, including hardware solutions from Cisco [cisco.com], F5 [f5.com], and Alteon [alteonwebsystems.com]. I'm not too familiar with Beowulf-type clusters.

If you are looking for software to manage groups of systems, that's a whole different story. You might look into Enlighten DSM [enlightendsm.com], Tivoli [tivoli.com], or OpenNMS [opennms.org]. I'm sure there's a lot of competition in that field as well, but I don't have any experience with those products.

Flight of the Condor

[scotfree]

I'm involved with a part of the Human Genome Project, and we run (around) 100 linux boxen under Condor [wisc.edu] for our often quite large computations.

This is not what everyone wants from a cluster; the focus is on large amounts of computation over time, not particularly fast computation over a few minutes. And it can take some work to break a task into jobs it will distribute effectively. Perhaps I should be clearer: it's intended to distribute jobs not cycles. And it can be hard to control how local file access and networked file access interact, if you need to. And it does some tricks with user priority which don't fit with some models of shared access.

Nevertheless, it works, and it's a wonderful thing to see a giant numbercrunch you've been running as one process split up and run in parallel, my goodness. And Condor has allowed us to do that without getting involved in any tricky parallel programming or even changing code: jobs run on their own box, happily unaware that they're part of a cluster at all, essentially.

So while I've had my issues with it, it has certainly been useful to us and is worth checking out. It's easy to find faults with anything; this has allowed us to get our work done, which is the main thing. I should also mention that Condor is continually improving, and the new version we've just installed seems to resolve at least some of our problems.

I'm interested to see what else is out there...