Kernel Fork For Big Iron?

Boone^ writes: "ZDNet is running an article on the future of Linux when used on Big Iron. Just a bit ago we read about running Linux on a large scale Alpha box, and SGI wants NUMA support in Linux so it can support their hardware configuration. The article talks about how memory algorithms used with 256GB machines would hamper performance on 386s with 8MB ram. So far Linus et al have been rejecting kernel patches that provide solutions for Big Iron scaling problems. How soon before a Big Iron company forks the kernel?"

Forks and the maintainer

Not really. Forks are also justified if the maintainer has effectively abandoned the project and refuses to relinquish it (and someone else has to "seize" control to make sure that it continues to go forward).

Just as importantly, forks are probably necessary when a significant part of the user/developer base disagrees with the direction of the project. This usually implies that the forked version and the original version are aiming at solving different problems within the same vein. If the original project wants to continue in the original direction and some people want to use the source to solve a slightly different project, then they pretty much have to fork in order for the project to achieve its maximal result of being most useful to the most people.

This isn't a bad thing if it's done right. It's just that most of the big forks you hear of are at least partially the result of bitter, angry wars (OpenBSD anyone?). You don't hear that much about the ones which are completely amicable.

Re:Let them fork

What I would believe to happen for the big iron machines is that they would have a different directory for them and the memory management code will be under there. So now you have the memory management for big guys and the other memory management code under the same source tree. When you compile your kernel it looks for the proper managemnt code. I know not that simple there is more to it than that but is what will happen if they fork and come back together.

Re:Why not?

on the subject of forking...
whyn do we need one huge kernel anyway? Probably several kernels are needed. One for big-ass servers, one for tiny-ass routers, one for mainstream workstations, and one TBD. Having one all encompasing kernel makes building the kernel a pain in the ass. I've been using linux for four years, and I still have to build my kernels a couple times before I get it right. So many freakin options, i'm bound to get something wrong.

but that's just my opioion...

-earl

Re:Why not have a kernel option...

They do this now for 1 Gig mem limitations. The problem is that there are so many #ifdef and #ifndef's in the linux kernel now that some people do not want added kernel options (more #ifdefs).

One of the issues that people seem to fail to realize is that Linus is not necessarily rejectiung the patches because of what they do, but how they are implemented. If patch code is submitted to Linus and the patch is going to make mataining that system difficult (read messy unmaintainable code) Linus will reject it. Linus also does not like large patches either. He likes bits and peices and clean fixes. Hey he started this whole thing, I think he has that right.

Another thing to think of is that ZDNet is a news network. Everyone has been saying that the kernel will fork and blah blah. There are already forks in the kernel but people just don't realize this.

Redhat kernels: Have you ever tried to apply a patch to a stock redhat kernel? I know that since RH5.2 they ship the Linux kernel with there own patches.

SuSE kernels: Last SuSSE I installed (5.3) had both a stock Linux kernel and a custom SUSE kernel with custom SuSE patches.

Corel: never tries them but they patched kde and made it hard to compile other kde software with there distro.

Point? There are already forks in the Linux community, yet it goes on. That is the whole thing about open source. There can be forks. If an idea is good it gets into the mainstream kernel. But these 'forks' need to be tried first and become tested and cleand up in such a maner that they can exist with the rest of the linux kernel.

If you think that everyone is running P200 or P500 or GigHz machines you are wrong. I am sure that there are lots of people out there that are running old 386 / 486 with Linux as routers firewalls, etc. After all you do not need a superfast machine for a firewall if all you are going to firewall is 3 or 4 other machines.

I don't want a lot, I just want it all!
Flame away, I have a hose!

Speak for yourself

I am constantly putting Linux onto old hardware. Need a quick, dirty, and cheap NAT box? Throw Linux on a DX2/66.

MP3 file server for the geeks in IT? Throw in a big drive, but a 486 will do.

Hell, my company's web server is running on a low end PII, and I think it's a horrendous waste! It could be doing *so* much more.

Linux is a UNIX for cheap Intel hardware first. That's where its roots are, and I don't see why it should sacrifice its roots for big iron that can quite happily run a UNIX designed for big iron.

Neither does Linus, apparently.

not too bad

So many things are distributed as kernel patches that it doesnt really matter. Anyone with that kind of hardware will obviously have the expertise and the money to install an appropriate kernel patch. No box that big is going to run an out-of-the-box kernel anyway, if you're using that sort of hardware, you're going to want to tweak it. As long as there is not a division in the majority of users' needs, there is not likely to be a major fork.

Ifdefs imply accepted patch.

(sorry for the double post, this is to the first half of the comment).

It depends on how pervasive the code changes have to be. If it involves #ifdeffing every single file, then it's going to be very difficult to maintain that, and it's going to be very unlikely that the maintainers of the project are going to allow that feature to remain part of the major distribution.

That problem is a dual-edged sword. It also means that maintaining one big patch is a complete nightmare. Every version of the kernel that comes out has to be separately patched, with two important considerations:

• The code which needs to be inserted has to be reinserted. If this is all separate files, that's easy, but if it's not that's a complete nightmare. And the code to call into that separate file is then a nightmare.
• Any changes which have broken the patch have to be investigated and possibly changed. If you're working on filesystem patches, for example, someone working on the core fs work may have broken your patch without your knowing it, because they're not including your code in their coding/debugging process. So every time there's a change to the kernel, you have to figure out whether that change will potentially break your work.

The only way to resolve the second is to keep the patch inside the actual kernel, so that the authors of the rest of the system are aware of it, and will either try their best not to break it, or will do first-round of changing the new functionality to work with their changes.

Basically, it comes down to how pervasive the work has to be. If it's a really pervasive change which touches on almost everything, then the only option from a software engineering perspective is a fork. Anything else is being done from a feel-good PR perspective, because it just doesn't make any sense from a technical perspective to try to maintain a huge patch that covers everything.

Re:Why reject?

Because now is not the time.
A great many features start out as independent kernel patches.
When thigns stabilize down the road, I'm sure they will gladly put 'Big Iron' flags in the compile stuff.

The point is, linus (et al) can't just stick everyting everyone submits, big OR small, into the main kernel, especially if it's not even developed yet!
Also... the feature set for current kernels is already listed... and this isn't one of htem.

You don't just add shit to a project partway through because someone wants you to.

I'm sure than by the time 2.5 kicks up, we'll see a 'big iron' flag in main kernel options.

ZDNet's tendencies to sensationalize at work?

So far Linus et al have been rejecting kernel patches that provide solutions for Big Iron scaling problems.

This makes it sound like Linus has been rejecting them because they provide solutions for Big Iron scaling problems. Having read kernel traffic and the linux-kernel list enough, this statement looks immediately suspicious. I have never seen Linus ever purposely reject a patch that's an all-around good fix for a problem. Usually it's "Well, Linus rejected my patch even though it does all this cool stuff and fixes all these problems, so it's probably because he just doesn't like such-and-such feature/platform/interface" and then Linus replies, "no, I rejected them because you're a dumbass and your patch sucked."

The link to the SGI page somewhat confirms this:

9. When will this code be added into 2.3?

Linus agrees in principle to take this code in. It has
already been reviewed by Ingo and Andrea. Linus wants to
clean up the page allocation data structures a bit before
imposing this code on top of it; I am trying to help him
do that. New: As of 2.3.31, this code is in under
CONFIG_DISCONTIGMEM.

I just kinda heavily doubt that Linus wouldn't want awesome NUMA support if the potential was there. My best bet is that the people pushing for it just aren't on exactly the same wavelength as Linus (is anyone?) and it's slowing down progress.

Another quote that points in this direction

Linus: "A lot of the problems, especially with NUMA, are that the solutions tend to add complexity that simply isn't needed at all on 'normal' machines,"

I don't think Linus mean any solution, just the solutions presented to him.

Re:Inevitable, but not so bad

It sounds inevitable that a Big Iron fork will occur, and as Linus says above, this is not necessarily a bad thing. The problem comes when you have competing factions trying to do the same thing and causing confusion (as in the UNIX wars of the past). But when you have different solutions for different problems, yet everyone is moving forward together overall, it should be manageable. Indeed, it should be helpful, for it maximizes the solution for each platform.

The biggest potential problem of forking an OS is binary and API incompatibility. The reason most people use computers is to run specific applications. I want to be able to walk into my local CompUSA/log on to Egghead and get a copy of application X and run it on my computer. I don't really care what the OS is, as long as it runs application X.

If I've got Linux on my system, I'd like all applications that run on Linux to run on my system. The more forks that introduce binary or API incompatibilites, the less chance I have of being able to run the apps I want, and the more reason I have for removing Linux from my computer.

If Linux wants to be a mainstream desktop OS, it needs to make sure it doesn't fork too much. That was a big part of the reason desktop UNIX failed to take off in the late 80's/early 90's.

Supporting 386s: Some Problems...

Yes, it is nice that it will still run on a 386, but there are other factors to consider:

1. Earlier platforms generally had no CD-ROM. Most Linux distros (except for fringe distros) come on CD-ROMs. Most people do not want to buy a CD-ROM for their 386, 486s. There are places that offer small "floppy-disk-sized" Linux distros, but they are obviously chopped. 1400K on a 500MB HDD.

2. Earlier machines usually had a 5 1/4" floppy disk, until the late 486s started really using 3.5" floppies. Most people are not going to spend money and time ripping out an old floppy.

3. Earlier machines had RAM limitations, aside from the fact that no one wants to really waste the money on putting more EDO memory into an obsolete machine.

4. Some earlier machines had fscked BIOSes, aside from Y2K-unfriendly BIOSes; Most people will not research whether the particular BIOS is okay to determine whether or not to spend money on the first three items.

5. Earlier machines had ISA, EISA, etc. Oh, what, you want to run GNU/Linux in something other than CGA?

6. Earlier network cards are not all supported to get around many of these limitations... I tried to get around not having a CD or a 3.5" floppy in an old 486 by using some sort of older ISA-based network card.

Obviously, there are many issues to consider before nodding one's head to allow Linus to try to preserve performance in ancient boxen for nostalgic purposes.

Lucas

--
Spindletop Blackbird, the GNU/Linux Cube.

Re:ZDNet's tendencies to sensationalize at work?

It's not excessive to expect someone to recompile their kernel to get optimal performance under extreme circumstances. It would be excessive to expect someone to recompile under tiny differences, but we're talking about the difference between 64-128 megabytes and 256 gigabytes of memory. People setting up machines that use such enormous amounts of RAM won't be put too much out of their way to recompile with a ENORMOUS_MEMORY option.

-----------

"You can't shake the Devil's hand and say you're only kidding."

Inevitable, but not so bad

From the article:

The process of non-standard kernel patches is just fine with Torvalds. "On the whole we've actually tried to come up with compromises that most people can live with," he said. "It's fairly clear that at least early on you will see kernel patches for specific uses -- that's actually been going on forever, and it's just a sign of the fact that it takes a while to get to a solution that works for all the different cases." He continued:

"That's how things work in Open Source. If my taste ended up being the limiting factor for somebody, the whole point of Open Source would be gone."

It sounds inevitable that a Big Iron fork will occur, and as Linus says above, this is not necessarily a bad thing. The problem comes when you have competing factions trying to do the same thing and causing confusion (as in the UNIX wars of the past). But when you have different solutions for different problems, yet everyone is moving forward together overall, it should be manageable. Indeed, it should be helpful, for it maximizes the solution for each platform.
________________

Not an issue

If you've followed the SGI/Linux debate on K-T, it's obvious that they indend to incorporate the option to enable BigIron features in the future, just not for 2.4 - as has been traditional with Linux.

Even in the cases where Linus has outright rejected BigIron patches, nothing stops a hardware vendor from patching the source after the fact - almost every major Linux distribution does this now for x86/ppc/sparc etc. (NFSv3 is a great example)

Re:Why not?

So Quake isn't the big issue here. Oracle is the big issue. As is Sybase and DB2, etc. The problem is, at what point will ISV's say "this isn't Linux anymore"? The whole reason that large companies like us (SGI) and IBM, et. al. are going to Linux is to get more applications. If we issue the SGI patch for moster systems, we could do all kinds of things like rearrange and add locks, add kernel threading types, and make the kernel preemptible. Is that really still the Linux kernel in the eyes of Oracle? Probably not. Then we lose 'cause customers aren't going to buy from us to run Oracle if they can't get support from Oracle (whether they will buy from us to run Oracle anyway is another question).

The other reason that we are scared of the monster systems patch is the number of Linux kernels that come out. How often do we recheck the patch? Which kernels do we release the patch officially for? How do we decide? There are no really good answers to any of those questions which is why the big patch is to be avoided if at all possible.

Inappropriate Ifdefs: BAD

If the system gets wedged up with a whole lot of #ifdefs, that makes it more and more difficult to maintain. LOTS of them can make software impossible to maintain.

I wouldn't be shocked if the stretching of boundaries that comes from:

• "Big Iron" changes, as well as
• Embedded System changes

winds up turning into there being some clear demands for forking.

The fundamental problem with a fork comes in the code that you'd ideally like to be able to share between the systems. Device drivers float particularly to mind.

After a 2-way fork, it becomes necessary to port device drivers to both forks, which adds further work.

And if a given driver is only ported to one fork, and not the other, can it correctly be said that

Linux supports the device

or do we need to be forever vague about that?

Re:The obvious solution: the kernel does have to f

Soo.. Bleh, I hate it when people talk about linux liek hes omnipotent.

Read KT [linuxcare.com]. Read KT [linuxcare.com] often.

Re:What's wrong with ifdef's?

Well you could put (the analog of) ifdefs into the Makefile. e.g. if there were big differences between conventional and big iron ways of doing things with feature 'foo', you would have 'foo-garbage.c' and 'foo-bigiron.c' and have make figure things out accordingly.

Of course then you would have to ensure that both offer a similar interface so that either can be used transparently. This *could* be a maintainence nightmare. I think there are a lot of ways that this *could* be done, but it depends greatly on the details involved whether it will be practical or not. I find it hard to believe that Linus would have looked over something as obvious as ifdefs or makefile tricks, so he's probably used his (undoubtedly god-like) judgement to decide that it would be a bad idea in the long run.

What's wrong with ifdef's?

I don't understand why anybody should fork code only because it has to behave differently on different systems. Why not use ifdef's? If too many ifdef's would be needed it may be better to have separate files and an option in "menu config". Even the current configuration system can handle it.

Forks are usually justified only if the original maintainer pollutes the source with hacks or changes the license.

Why reject?

Why are the patches being rejected? Couldn't they be conditionally compiled in?

There is a patch out there that stores the computer's state to disk before shutdown and then give you an instant boot. My home machine is used that much, and my UPS needs repair. This patch would be useful for me, but I'd have to patch it in by hand and then I'd be out of sync with the official Mandrake kernel. That means I'd have to patch in security update by hand.

The problem is, this patch is as useless to Big Iron as support for 256GB of memory is to me (right now). But why can't both Big Blue and I have our way with conditional compiles? All it would take are a couple of more menu selections in xconfig.

Do you have more that 2G of memory?
Would you like instant-on?

hmm..

Perhaps the time has come to fork the older machines.. Few of us run Linux on anything less powerful than a Pentium, and even fewer on a 486.

I don't know, it depends on where the split of cost/benefit falls.. ZD doesn't say...

`Sides, having a Compaq/SGI/IBM 'approved' kernel patch doesn't hurt much..

It only makes sense

I'm not claiming to be a kernel expert, but forking the kernel so that there would be kernels specialized for specific applications only seems logical. A builder doesn't go around hammering everything in site because the hammer obviously isn't the correct tool in every situation. It's great for pounding nails into 2x4s, but isn't so good when it comes to painting walls.

Specialized kernels are good, so long as the support behind all of these kernels remains great enough. I don't think I need to point out the possible pitfalls of forking the kernel and thus, effectively forking the developers behind the kernel into two or more camps. But at some point, the linux kernel that runs on a 386 should be different than the one that runs on the XYZ super computer, just because it can take full advantage of all the wonderful scaleability that the XYZ super computer offers.

Anyway, as I said I'm not an expert but this just seems logical.

Kernel fork for big iron? Why not?

There's kernel "forks" for hard (deterministic) real-time (RT-Linux, etc.). There's kernel "forks" for non-MMU machines (ELKS, uCLinux, etc...). So, why not a "fork" for big iron? If the fork for big iron doesn't hinder current modern machines or improves overall operation- it will become the main fork with the one that just supports the older machines becoming like the other "forks" we see today.

Re:Supporting 386s: Some Problems...

Let me clear up a few things here.

1. Earlier platforms generally had no CD-ROM.
Install via NFS or on a pre-formatted hard disk with all the necessary files. Been there, done that.

2. Earlier machines usually had a 5 1/4" floppy disk, until the late 486s started really using 3.5" floppies.
You can boot from a 5.25 floppy disk as well as from a 3.5 one. Besides from booting for the installation, there is no need at all for a floppy drive.

3. Earlier machines had RAM limitations
Many old 3/486s can use up to 16 or even 32 MB RAM. That's more than enough for a small (slow) home-sized server. Even 8 MB does the job.

4. Some earlier machines had fscked BIOSes, aside from Y2K-unfriendly BIOSes
Y2k is only an issue during boot-up, after that you can set the system's time to whatever you want. From what I've seen, Linux deals better with really old motherboards than some brand new ones.

5. Earlier machines had ISA, EISA, etc. Oh, what, you want to run GNU/Linux in something other than CGA?
There are very good SVGA cards for ISA, although running XFree with a "modern" window manager on such an old box is suicide. However, any kind of video card does the job for a "server" type of computer.

6. Earlier network cards are not all supported to get around many of these limitations
Granted, very old ISA cards might not work well, but many cards do. NE2000, old 3Com cards? No problem, work fine, and deliver good speeds too.

To make a long story short, killing support for old systems is a Bad Thing IMHO, and isn't necessary either, it would only make the kernel tarball smaller. I'm all for conditional compiles, and I actually wondered why some of the kernel patches out there (like the openwall patch) haven't been put into the mainstream kernel as 'make config' option. If they can put in accelerator thingies for Apache, why not this?

A brief history of computing.

Okay, first there were systems, and the were all different.

Then someone "abstracted" them with "BIOS"

Then there were lines of systems, and they were all different.

Then someone "abstracted" them with "C"

Then there were platforms, and they were all different.

Someone (Transmeta?) will come up with a way of abstracting platforms (or architectures) and
make them "seem" the same.

This relates directly with performance increases. When you find yourself wondering what is going to make a 10GHz system better than a 1GHz system I think the answer is the level of abstraction.

Any number of quibbles can be made with the above statements, but I am illustrating a point, not
being a historian.

-Peter

There is a point: One size rarely fits all.

This was bound to happen sooner or later. The Linux kernel's flexibility is being taken to the limit, and people are forgetting the easiest way to improve performance for their particular rig: Customize your kernel! You can add all the code in the universe, and then you pick and choose the particular things you need or don't need! Say I run a 486/25 with 16 MB RAM as an IP Masq router. The hard drive is an old IDE with 600 megs of space. I have two network cards, and that's about it. Do I need SCSI support? Do I need to support joysticks, X, Pentiums, AX.25, or anything else? No! I compile a kernel specifically to run the IP Masq, and run it well. My P100 laptop, on the other hands needs a bit more. I use it for packet, so I need AX.25. It uses PCMCIA, so PCMCIA support needs to go in. I use XWS to run Netscape and the GIMP, so I need graphics. But, my HD is not SCSI. I yank out SCSI. My CPU is subject to the 0xf00f bug, so that gets included. I brew a custom kernel, and boot time is a lot shorter. My big-rig is a C433. I need just about everything, as I have a 3dfx card for Quake3; XWS; a SCSI scanner; and a connection to my Packet base station. I optimize compilation for the higher-end computers. I plan on getting a Cube from Apple and putting SuSE on it. Again, by optimizing the options I optimize my system. Get the point? If you want a once-size-fits-all kernel, use Windows. If you want a kernel which can be adjusted for your particular and peculiar environment, use Linux and customize your kernel! Now, for my laptop.

Re:Inevitable, but not so bad

Uhmmm....There would be a few problems:

1) Is the resulting code still Linux?
This is a BIG question, especially for IBM and SGI who want to say they're Linux supporters. If Linus doesn't grant use of the Linux name to their OS, they're back to naming the resulting kernel something other than Linux. Big PR problem.

2) Will the "Linus approved" patches make it into the follow up kernels released by IBM and SGI?
I'd be willing to bet both companies are willing to do the right thing and include them, but how big can this fork get?

Now, all that aside, distros have been doing small scale forks for a while now. I think SuSE had a 1GB mem patch, and RedHat frequently patches the kernels they distribute. Nothing bad for most ussers.