One-Machine Linux Cluster

An AC wrote: Forget Beowulf ^? clusters, Jacques Gelinas has made available a kernel patch to enable many virtual servers running on the same machine, even the same kernel. Read his original message posted to the Linux kernel list." Imagine what this will mean for hosting companies...

Everyone gets their own psuedo server

haha.. better read the fine print and make sure you actuallu get your own *computer* including box and powersuppy, and motherboard-- or you may end up sharing your box with 100 other ppl :-P

Very Useful

Slashdotted before I could read the whole thing. :( But, as a sysadmin for a smallish web devolopment/hosting company I could REALLY use some separation between certain clients. Sure, this isn't ready for production systems but one day it may be.

The patcher is right...modern CPUs (for my industry) have PLENTY of power. What I hate is having to run some third party app for a client (even in a Linux environment) that *might* affect the whole machine. This patch holds the promise that I won't have as much to worry about.

Yes, this is a good thing.

Basically Like OpenVMS' Galaxy?

Basically the same idea as Galaxy. Check it out for ideas.... http://www.openvms.compaq.com/availability/galaxy. html

Beaowulf not the target audience

This has just about zero to do with clustering, if anything this is the opposite of clustering. However this IS very very interesting for Web Hosts and just about anyone else that wants to create and maintain multiple environments for developement, test, etc. Image, being able to carve up a mid-range machine like you can an S390 (or other Mainframe class machine Like Sun's E10/15K). So suppose IBM takes this an runs with it. Linux is already ported to RS/6000 and AS/400, now you could get 8 processors of RS/6000 goodness, run production on 4 processors, Test on 2 processors, and Dev on 2 processors.

The devil will be in how you refresh test and dev from production, but that can probably be done inside Logical Volume Manager.

This is very very cool stuff it will be very ineresting to see how it stacks up against the big boys in Virtual machine space.

Re:Beaowulf not the target audience

now you could get 8 processors of RS/6000 goodness, run production on 4 processors, Test on 2 processors, and Dev on 2 processors.

What you're suggesting is pretty much the opposite of how this package works. As the author himself states, you cannot dedicate hardware resources to a vserver. Only one kernel is ever running, and you use all of your cpus or none. Process- and user-space isolation is provided, but if a process in one vserver tickles a kernel bug that crashes the system, the whole ball of wax will come down with that vserver. (Likewise, it's very likely that a kernel-level root exploit will allow you to break out of the vserver and attack the whole system.)

Essentially, vserver is to the process space what chroot is to the filesystem layer.

This is not inherantly better or worse than the "system partitioning" approach; it's just a different approach, and will have different uses.

Re:Beaowulf not the target audience

IBM is already running 15000+ linux servers (seperate kernel and all) on a single iron ..


They're running that on an iron? My god, technology is moving so fast now. They've skipped right over the toaster.

Isn't this sorta the opposite...

... of clustering. Its... slicing your box up...

User Mode Linux?

Can anyone tell me how this is different than User Mode Linux [sourceforge.net]?

Re:User Mode Linux?

User Mode Linux is basically a VM. It uses virtual devices for hardware multiplexing. Read the "Alternative technolgoies/Virtual Machines" and "Alternative technologies/Limitations of those technologies" for why this is a different (and better in some instances) solution.

The vunify tool has significance when differentiating between VM's and this.

Re:User Mode Linux?

At the moment, User Mode Linux does separate the processes in a VM from the host system. That's because the kernel image itself is writable for the processes running in a UML virtual machine, which means that processes can break out of the virtual machine pretty easily and gain access to the account running UML on the host system. In addition, even if this is corrected (perhaps it has been during the last few weeks, I haven't checked), the kernel memory would still be read-only for the processes run by it, so different processes in the virtual machine could snoop each other. This means that User Mode Linux is great for testing stuff, but it only moderately increases security.

The patches for compartmentalization which mimic FreeBSD's jail(8) feature are completely different. If they are done properly (and checking this will require some time), they can provide complete separation of the processes running in different compartments. Performance is probably a bit better, too, because only one kernel is running, and not a stack of two.

Again, if you need compartmentalization now, and you have security concerns, you should either use FreeBSD, or GNU/Linux on S/390. This new kernel feature will need a bit of time to settle down and work correctly (from a security point of view).

Actually this is kind of an old idea

Hostpro, now Interland has this sort of thing for freeBSD. It used to be called vserver. The new improved version is called Freedom. It's been out for years.

mosix

I wonder how this would work with mosix [mosix.com]... it could be a dream system!

You could use mosix to combine the compute resources of several boxes to look like one box. And then, you could use this divy up the space so that people don't step on each other. When anyone (working in thier own space) kicks off a large compile, the load would transparently be distributed among all the boxen.

Of course, I have zippy experience with any of this, but it sounds possible.

It's not the processing power

It's the control over it.

Mainframes have insane amounts of control over user processes (a Linux image essentially becomes same), as well as the ability to allocate more resources, fewer, provide fine-grained process accounting, shut down processes, migrate them elsewhere (part of the IBM dataceter Linux concept is the ability to migrate nodes around the country as needed).

What a mainframe doesn't have to offfer is insane amounts of processor power or memory. Disk, and disk I/O are quite another matter -- the amount of aggregate bandwidth a z390 has to offer is impressive.

PC-based virtualization clearly has some advantages, through not all of those offered by a mainframe. A rack of virtualized PCs probably does offer a higher processor density than the equivalent mainframe, however.

Resource limits are needed by hosting companies

My particular interest was to find virtual hosting solutions that would (1) not allow one runaway virtual server to deny the others of at least a predefined minimum level of CPU, RAM and I/O (disk and network) resources and (2) give any one virtual server extra resources if they were available. From my reading of other slashdotter's posting and the info on the web I've summarized below the various virtual server hosting solutions mentioned. Someone who actually has used these products should actually correct me.

Linux can natively be configured to enforce disk quotas and (with more difficulty) [linuxdoc.org] manage network bandwidth [linuxjournal.com] without any special virtual server software. Also the native unix process scheduling algorithm does reduce the priority of CPU bound tasks. The getrlimit(2) system call can be used to set various limits per process (not per virtual server unless the virtual server runs as one process I guess.) I know of no way to specifically limit disk bandwidth on Linux.

Freeware such as s_context [solucorp.qc.ca] and user mode linux [sourceforge.net] provide no control over how much resources one virtual server gets over another besides disk usage. Other limited resources like CPU, disk and network bandwidth (RAM?) are shared just like they would be shared by separate processes under a single Linux system.

FreeVSD [freevsd.org] is not a virtual server, but a collection of scripts, binaries and multiple copies of hard-linked read-only filesystems for the common system environment. It is has the best chance for winning the total performance award but has no extra features for resource limits between systems.

True virtual machines. (E.g., vmware [vmware.com]) provide very good isolation, but this leads to little sharing of excess unused resources between virtual servers I believe. They also have poorer performance in general because so much emulation is done.

The commercial, proprietary Private Server [ensim.com] product from Ensim [ensim.com] seems good from the marketing blurbs which say that they have "their own guaranteed share of the servers resources, including CPU, memory and bandwidth". I wonder what the performance penalty for this is and how much does it cost? Can anyone comment?

*This* is why open source works

Much respect to this guy. He's taken something thats big, hairy and complex and looked at it from a different direction. Because he's got access to the source he's been able to do something novel with it in what appears to be an efficient and simple way...you couldn't do that with any of the closed source OSes out there today!

The beauty of this is that there's *one* kernel running so, apart from any overhead of selecting the environment, you pretty much get the same performance as running native. This has got to have 1001 applications.

One of the things I'd personally like to see is some kind of overlaid filesystem so each image by default gets /bin /lib etc. from a generic set but users can modify them if they need to - this would allow a sysadmin to keep the default system current while not preventing 'owners' of an individual image from being able to change things if they need to....I vaguely remember something like this for CDs - anyone got the details? Time for a bit of experimentation ;-)

CPU Time limits and scheduling

I wonder if it would be practical to associate absolute CPU time limits or CPU usage percentages with a security context id in order to prevent a certain security context from hogging all CPU ressources.

A similar thing would be desireable for resident set size (real RAM usage) and virtual size (process size) per security context.

Re:bah

Well hey, on a multiprocessor machine it might be interesting. Dedicate a processor for each node and you have four beowulf nodes in one box.

Not only would it be cool for developers to test Beowulf-enabled code, but it would be awesome to have each node independantly accessible from the network.

Re:Someone doesnt understand the reason for a clus

Someone doesnt understand the reason for a cluster

I think that was whoever wrote the headline. This doesn't buy you what a cluster buys you, which is more MIPS and RAM working on the same problem. This buys you multiple relatively independent environments on one machine. Hence the reference in the /. article to hosting companies. This is like the Sandinavian ISP that replaced a bunch of Sun boxen with a much smaller number of larger IBM mainframes. The mainframes run IBM's VM (stands for Virtual Machine), which is roughly analogous to the "root-kernel" dicussed in the linked-to articles, then runs many independent copies of Linux under VM. Each user gets his own copy of Linux to screwup^H^H^H^H^H^H^H do as wishes with. VM 'fools' the copies of Linux into thinking they each have their own machine. Here we have Linux replacing VM, so now a Free product can do some of this (assuming it works, which, being brand new, I wouldn't bet my income on. But I'll bet it will work well soon.), though not all: VM lets you run multiple OS's on the box, which this doesn't. But it does a lot - separate root logins/passwords and process lists for the 'vurtual Linuxen', for example.

Re:wow

(Goodbye, karma.)

I know this is completely off-topic, but here it is anyway.

Your understanding of the prefix "meta-" is incomplete. In addition to indicating syntactic self-reference (see Hofstader), it can also indicate semantic self-reference (see... well, Hofstader; he talks about this, too, in his discussion of GOD: God Over Djinn).

SGI has a device for connecting crossbar routers together to form large single-system-image computers. It's called a metarouter:a router for routers.

Likewise, a cluster of clusters would be properly called a metacluster. Since "Beowulf" is commonly synonymous with "cluster," the term "meta-beowulf" is pretty much correct, even though it makes me cringe.