Linux Kernel Back-Door Hack Attempt Discovered

An anonymous reader writes "The BitKeeper to CVS gateway was apparently hacked in an attempt to add a root exploit back door to the Linux kernel, according to the linux-kernel archive. The change was in the file kernel/exit.c and changed the user ID of a process to root under the guise of checking the validity of some flags. The core Linux BitKeeper kernel repository was not at risk, and in fact it was the BitKeeper CVS export scripts that detected the unauthorized modifications to CVS. The changes were falsely attributed in CVS to long-time Linux developer davem (David Miller). Users of the BKCVS repository should resync their trees to remove the offending code if they had replicated it since yesterday."

Well well

Good to see the system works. You would wonder what would happen if said hacker was working for a company on a similar closed source program. Would it have been detected?

Re:Well well

Good to see the system works. You would wonder what would happen if said hacker was working for a company on a similar closed source program. Would it have been detected?

You mean like Borland's Interbase? The compiled in backdoor [cert.org] wasn't discovered until after the database opensourced.

My favorite quote from the advisory is:

"This vulnerability was not introduced by unauthorized modifications to the original vendor's source. It was introduced by maintainers of the code within Borland. The back door account password cannot be changed using normal operational commands, nor can the account be deleted from existing vulnerable servers [see References]."

How long was it in there? "These security holes affect all version of InterBase shipped since 1994, on all platforms."

The advisory dates from 2001 -- you do the math.

Re:Well well

Hmmm..but would they have even found the security hole if it hadn't been open sourced?

Re:Well well

Backdoor [google.ca] to [cert.org] windows [cert.org] not [cert.org] due [cert.org] until [cert.org] Longhorn [cert.org] ? [cert.org] ? [cert.org]

Re:Well well

All of the vulnerabilities I listed made it into official releases before being patched. The bug this story is about didn't make it one day in the source tree, let alone into an official release.

Sorry about the Protegrity one, I must've linked the wrong one. I was looking for this [cert.org] one (the one exploited by the slammer worm).

You mean, "what's really gonna bake your noodle...

... is would the code be exploited if nobody had said anything?"

Re:Well well

You would wonder what would happen if said hacker was working for a company on a similar closed source program. Would it have been detected?

Well the 12 backdoors I put into the Windows XP kernel haven't been detected yet.

Re:Well well

Although I see where you're going with this, I think a lot of people might ask whether this shows vulnerability in OSS instead. Sure, you and I appreciate this as a validation of the system but is that really how the media is going to portray it?

All I'm saying is that I certainly won't be surprised when closed source vendors start using this in their anti-OSS campaigns.

Re:Well well

Yeah. There's no way they'd want to show an open-source project finding and warning about a problem within 48 hours in a positive light.

They'd find their customers wanting timely patches and accountability.

Re:Well well

Good to see the system works.

And what if we just haven't discovered the code that got through yet...

You've got to ask - assume nothing.

+5, Tin-foil hat.

Re:Well well

Well, I guess that means all the closed source developers have the same problem. And I guess they probably don't know either.

Re:Well well

> And what if we just haven't discovered the code that got through yet...

1. We know that SCO have been looking very closely at the Linux source code.

2. We also know that none of the Linux boxes which serve major anti-SCO websites have been hacked into.

3. We can deduce therefore that SCO have not found any backdoors in the Linux source code.

While given their general level of (in)competence this doesn't amount to proof that there aren't any, it's probably a fairly safe bet.

Re:Well well

It was only detected because software found a discrepancy.

This would happen at any closed-source shop that had the same software.

No human eyes discovered the problem, and if someone hadn't installed the checks, it might not have been discovered for months or years or ever.

Re:Well well

Not only that, but imagine this. The hackers (in the real sense, not the TV-movie sense) who write the real low-level stuff that makes various OS's work - for example in Linux people like Alan Cox, Linus, RMS, ESR, davem, and the other regular kernel contributors submit a lot of code. Well, those people dont necessarily but a lot of code is in the kernel.

Can anyone tell me for 100% certain that between GCC, the kernel, and various compile chain tools there isn't a subtle backdoor that creates an overrun, or a weak key, or anything like that somewhere along the line? Maybe what looks like an innocent bug or flaw or even stylistic change in one source combines with a similiar item in another source to create an exploit or a weak scheme.

These people - real hackers - are so clever (I mean serously, writing and maintain an OS for fun puts these programmers in the top 1% of all advanced systems programmers) that what is to say that they couldn't dupe everyone even with the source available to all?

I can imagine a situation where a corrupted/corruptable individual works hard to gain legitimate comitt access to certian tools that are widespread. GCC, the kernel, a shell or two, OpenSSL. That person starts making small changes that when aggregated expose a large exploit but when examined piece-mail are completely benign, or even benficical.

Does anyone doubt that its technically possible? How could any automated system or person ever discover this? I am a fairly competent programmer in some areas and there have been numerous times that I've had to dissect large pieces of code painstakingly over the course of days or weeks to trace back a nasty bug. Can anyone say that its not possible that this is *already* happening in the OSS world today?

Re:Well well

what is to say that they couldn't dupe everyone even with the source available to all?

You mean like this [acm.org]?

Re:Well well

As anoth poster has pointed, this is a well-known thought experiment, and what it boils down to is: at some point, you have to trust someone.

Have you audited your motherboard BIOS? What about your network card - how do you know it doesn't have an IP stack on the ROM that dials home and dumps your network activity to someone? Hubs? Switches? Routers?

Do you really know what lives in your hard drive controller?

Re:Well well

Leprechauns.

Leprechauns live on my hard drive controller, and spin it with all their tiny might.

They're like little green DJs when I use my RAID.

Re:Well well

RMS *must* be a low level hacker. He shows all the signs: long beard, long hair, smells, can't behave in front of people, yawns at conferences. Heck, he even farted at a conference at my Uni.

If he isn't a lowest level hacker, my world foundations are crumbling...

Re:Well well [Thompson: Reflections on Trust]

Of course, at some point, we do have to trust someone.
Ken Thompson [bell-labs.com] wrote an original speculative essay [acm.org] on this for CACM [acm.org] back in 1984 of all years.

It is really well worth the read. The short form is that there exists a way to subvert the compiler such that it is no longer trustable and it will build a back door into the OS forevermore. This paper is a must read.

Re:3 cheers for monolithic kernals

You honestly have no idea what a "monolithic kernel" is, do you.

My God! It's full of stars!

1 x 4 x 9

That monolith... oh... kernel.... right...

Re:Well well

Kinda proves Steve Ballmer's comments about the lack of security in Open Source development, doesn't it?!

No. I just proves you're a posturing idiot. The crack was detected as soon as it was attempted to be inserted, in the experimental development version of the code that hadn't even made it into any final distributions yet.

And here's another example of your idiocy:

If it happened in a software company, the hacker would be fired and probably charged with some kind of "espionage" charge and arrested.


This wasn't an "inside" job. If this happened at a company, to fill the analogy, it would have been an external person, NOT someone they could fire.

Re:Well well

Yeah, but anybody who feels the need to use stuff like this, probably updates often and checks stuff as a matter of course anyway, and possibly even sandboxes test kernels - so the damage is self-limiting. If you always want the sharpest blades, you have to understand you can cut yourself. Ordinary mortals mostly run stock kernels, from their distributor or kernel.org. Somehow, I can't see such an obvious exploit finding its way into a major distro.

And really, it's just more evidence that the Open Source model works. There is really nothing wrong with making a mistake, as long as you learn something from it and share what you learned with other people so they don't have to make the same mistake. Pretending you never make mistakes is another matter entirely .....

Daaaammmmmnnnn..

Someone has some damned big balls to do something like that...

Let's hope they're cut off.

Re:Daaaammmmmnnnn..

Why?

What's the penalty under the law for putting a backdoor in an open-sourced software project?

None.

That's it. That's the list.

Re:Daaaammmmmnnnn..

Our friend the DMCA, of course. Becomming root with wait4(3<<30) could clearly be used to copy files to which you don't have read access, thus circumventing an effective copy control mechanism. Heck, I'm a criminal for telling you that.

Seriously, though - there are probably many laws by which it would be illegal. The cracker gained unauthorized access to a system and he vandalized data. And the obvious intent was to create a backdoor in many more systems. If they find this guy, he'll be in serious trouble. The guy he pretended to be could probably also sue him for something.

Microsoft

Anybody point fingers at Microsoft yet? SCO?

Re:Microsoft

No, but I'd like to see them claim copyright infringement on back-door code.

I wonder why not a remote root hack

The vandal who put this in the CVS code tree obviously had a lot of skill.

It's a clever backdoor, and might have gone unnoticed, if not for those those good automated checks in the BitKeeper-to-CVS gateway. Notice that the particular coding style is a common C gotcha (using "=", assignment, instead of "==", comparison). At first glance it looks like the value of uid is being compared with 0, when in actuality it is being assigned the value of 0: root! The gcc compiler is good about warning for this, except that this too has been defeated: as mentioned on the mailing list, notice the unusual high number of parenthesis around this expression. That high number of parenthesis has the effect of suppressing the gcc compiler warning.

So, whoever did this obviously knew what they were doing and tried to obfuscate it. As somebody else mentioned on the kernel mailing list, if somebody is going to put in a backdoor like this, why not make it a remote root hack?

As it is now, the above hack is only locally exploitable. A process on the local system still has to call the wait system call with that particular combination of flags, in order to trigger the exploit and get root. To my knowledge, no known applications do this, because the combination of flags is supposed to be invalid.

If a spammer or somebody else was trying to backdoor the Linux kernel in order to gain a large number of machines to infest, then one wonders why they didn't put in a remote root exploit. It seems strange to go to all the trouble. Since this backdoor attempt has been caught and blocked, security will now only become tighter, and they might not ever get another chance like this.

Maybe it was intended to be used with another application, also backdoored in the same manner? It might be insightful to scan other open source applications and search for this particular usage of flags to the wait system call.

In any case, I'm glad this hack was caught!

Re:Microsoft

The actual lines of code and the method by which they got there were far too clever for either Microsoft or SCO. In particular, it looked like a check for an invalid combination of flags by root, but would actually set the process to root in the case of the invalid combination of flags (and the error return value would be overwritten).

The intent was probably that a CVS user get the bad version, work on other stuff, and send the diff (including the bad lines) to a maintainer in an otherwise good patch. However, the BKCVS gateway got confused by someone other than it changing the CVS, and complained, and Larry McVoy pointed out the issue, someone asked what the lines were, and other people figured out what they'd do. Now, of course, if someone had gotten that bit accidentally and submitted it to a maintainer, they'd notice, so the attempt seems to have failed.

Linus pointed out a benefit to using BK: even if the official BK repository were changed, he doesn't pull from it (because his local copy has all of his changes), and he would get an error the next time he pushed to it. The repository that would have to be attacked is actually his local disk, behind a firewall and not set up for anyone else to access at all.

If RMS wants to rant about revision control systems, he'll need to say that CVS needs to be replaced with a more functional alternative (Subversion, perhaps), not BK.

Re:Microsoft

The actual lines of code and the method by which they got there were far too clever for either Microsoft or SCO

It was a subtle change but I think it would have been caught if it had been submitted to Linus. He does review code and often catches mistakes. In this case assignment was used in a condition. To good C programmers this is bad taste. I noticed that right off and I haven't written a line of C in about 6 years. Linus isn't just a good C programmer. After half a decade of watching him catch stuff like this in just his public LKML messages, I'm convinced he would have seen this if he were reading braille hardcopy of it from across the room while drunk.

Re:Bad News

It's the quiet ones you've got to watch...

Yes, everyone who's upset about exploits they haven't heard about, raise their hands...

Yet another reason to use open source software

This is the reason I trust open source software. The power of peer review (in one form or another) catches these kinds of things before they are sent into the wild.

!!! rag

Imagine if this had sneaked into some Longhorn code right before shipping. Many eyes make few mistakes.

Re:!!! rag

No, you don't understand. This exploit was disguised as error checking code. It'd stick out in Longhorn like a sore thumb.

Re:!!! rag

That's right, if it was just something like:

/* need to be administrator here for temporary */
/* hack, must remember to change it back again */
/* later */
set userLevel = administrator;

Nobody from MS would have batted an eyelid.

hmm

Sounds like a plan to get the dirty GNU/hippies to upgrade to the real BitKeeper instead of using the communist CVS gateway.

That McVoy is a smart one!

Did you know his programmers need to feed their families and pay their mortgages? Very sad situation, I hope everybody buys 10-15 licenses ASAP.

more reason to sign patches?

Why not just establish a web-o-trust and sign patches?

That way people who hack in won't be able to send in signed patches to the system [e.g. even if they physicially update the tree others can trivially spot the unsigned patches].

That would of course, require people to actually think about security in terms of "oh sure people won't hack it because it hasn't been done...much...before."

Tom

Re:more reason to sign patches?

Yeah it's really easy to sign code when you don't have the key. All you have to do is try about 2^1024 different combinations of 1's and 0's. Simple, but not if you want it done before the Universe is gone.

And if you have access to the key, remember that it's encrypted with a passphrase. Assuming it's 40 letters or longer (Something like "This is a passphrase that is long but easy to remember. I would just like to tell you, Mister Password Prompt, that nobody will guess this!"), you would have to try about 100^40 different passphrases. That's hard.

So basically, it's really hard to forge a digital signature. Harder than breaking into the BK server, anyway.

Curious abot the hack, was it remote?

i want to know if the hack was a remote backdoor or "only" a local root compromise. In order to how bad was the hacker that try to do this.
Thanks to the admins and developers that detect that!

Re:Curious abot the hack, was it remote?

As noted on LKML, (current->uid = 0) was probably deliberately surrounded by brackets to avoid a gcc warning of an assignment within a test.

I'm not so sure about that. Personally I would have put the brackets there even in case of a normal test. They might not be necesarry, but I trust brackets more than I would trust my own ability to remember the precedence of every operator in C.

Re:First of Many?

Will this be the first of more kernel backdoors, now that the idea is out there?

Isn't the pertinent question... was this the first?

Alright....

I'll call ESR, he's got the guns.
You guys get Linus and make sure he brings Tove, since she could probly kick all our asses.

Once thats done we'll Larry McVoy, by this time hopefully he will have the IP of the slimeball.

The Pose rides at Dawn, we can kill some Trolls along the way.

So how do we know that there is only one?

Ok, so the scripts caught an attempt to install a back door. Everybody jumps up and down and sings the praise of the mighty Open Source Movement.

What if a backdoor was installed last week, or last month, but was not caught?

The fact that this was possible once, should really make people think about the possibility of it happened ALREADY, and determine if it is necessary to hunt through the code for a systematic review.

Instead, all we get is Microsoft Bashing...
Ugh

yet another reason for (CONSTANT == var)

In my code I always put the constant on the lhs so that the difference between the equality (==) and assignment (=) operator are caught by the compiler by accident.

if ((options == (__WCLONE|__WALL)) && (current->uid = 0))

In this case, it would make an attempted root hole more visible, as (0 = current->uid) would not compile.

The more eyes...

> Setting current->uid to zero when options __WCLONE and __WALL are set? The
> retval is dead code because of the next line, but it looks like an attempt
> to backdoor the kernel, does it not?

It sure does. Note "current->uid = 0", not "current->uid == 0". Good eyes, I missed that. This function is sys_wait4() so by passing in __WCLONE|__WALL you are root. How nice.

And this is exactly why folks should insist on open source code.

Assuming it was noticed, and I have little reason to think that modification of a project's cvs tree would go unnoticed, a closed source product would have to go up and down the development chain of command. Then likely up and down the marketing chain of command while a decision was made whether to say anything about it (yeah, right) was made. Meetings would be held, blame would be assigned, and - oh yeah - a discussion about a fix would ensue.

Perhaps I exaggerate, but only a little.

I remember when a beta of a game [unnamed software publisher] was working on got ripped off our company ftp site and passed around. There was so much hype about our game that the leaked late beta was a serious disappointment and effectively killed the good buzz the marketing folks had whipped up. [It blew anyway, got shredded by the gaming rags, had a lot of potential but an inexperienced crew and very little financial support.]

Of course, this situation is nothing like that.

There's always going to be someone trying to backdoor the linux kernel, windows, osx, apps galore. Having the source on-hand to look at gives you that added level of confidence that "hey, worst case we can fix it - deal with it ourselves" rather than go through the denial, silence, lame excuse, patch cycle you go through with closed source products.

Ebay-style attacks

While this attempt was thwarted, it makes you wonder though if someone could do an Ebay style 'attack.'

In other words: 1) Work on the code for a long time, developing good features and build up virtual reputation points so that people trust you. 2) One day decide to insert your backdoor amidst some big checkin. 3) Disappear.

It doesn't seem hard for someone to pay some random third world programmer to do this so. For example, if Red Hat had a guy in russia doing this they could, after the latest kernel was widely distributed, use it to attack Novell/SUSE.

Calm down, calm down...

Keep reading the thread - you'll read a bid of Linus, and a comforting explaination [iu.edu] of whats happenin' back there.

Trusting Trust

The Ultimate Backdoor, if anyone hasn't read about it:

Reflections on Trusting Trust [wbglinks.net].

You might want to doublecheck that gcc code you're compiling the kernel with...

As noone else seems to have pointed out yet...

The "backdoor" that someone attempted to submit was a local privilege elevation bug, not a remote compromise.

doubters are forgetting the foundations of OSS

I see some people posting some negative replies, or lamenting on OSS process, etc and saying how it didnt work, or how a psuedo-trusted patch would get in, if it went through proper channels, or some such crap.

this couldn't be further from the truth, you are all forgetting many things, #1 - the checking scripts run daily now, and Larry has mentioned he's going to step that up, still fixed within 24hrs is a damn good response time! closed-source could never be this fast.

#2 - all this talk of peer review, saying it didn't catch this or whatever nonsense, yes in a way it did, and whats more it's exactly what will keep semi-valid attempts or those through "proper channels" out of the code. You forget, millions of people around the world review this stuff, and someone, somewhere will find it relatively quickly, and not just because all the good developers (which is most of the millions) really LIKE linux and do their utmost to protect it, and ensure that no twits do things like this.

on the oft-side billions to one chance someone does something stupid like people said hire someone to do good patches for a long time, get trusted, and submit a patch with this kind of code in it, well, first of all, this is just stupid, it would take years to get that trusted from "zero", second, even assuming all that, the code would still get caught very quickly.

Like I said, someone, somewhere is gonna notice real quick, because the millions of us out in the world really happen to LIKE linux, and protect the kernel most of all, and I'm sure as the code worked its way into the tree, one of the people would catch it, and I'd be willing to bet several would see it at the same moment, including Linus, et all.

You simply can't pull a fast one over the great coders we have, these aren't your average coders, and remember, not just them, but all of us, really, in a way, put our heart and soul into supporting Linux, its a confidence we dont share lightly, the kernel is the most protected of it all, yes, for obvious reasons, its the most critical code.

But even outside the kernel, remember millions of people around the world are reviewing code 24hrs a day, every day, and posting notes about issues, patches, etc.

It's simply much harder to get by all that. Like I said, and I'll say it again, someone's gonna notice, and probably LONG before it even gets into the main BK tree, because even those reviewers ain't slouches!

Closed source has a smaller review team, and I know for a fact internal developers add back-doors to code all the time. I know many closed-source coders (not necessarily personally) that as a matter of habit throw in back-doors into every piece of code they write, because they hate their job, and the people they work for, and hate the product. Since very few people ever review the code, things can sit there indefinately and never get found.

remember this is a work of pride, something the community really cares about, we really want to see it succeed, and not have the issues like this, or that others have, we want to protect it at all costs, in any way, to ensure a good future, and protect the users out there.

remember, we're users too! If it means that much to you, wouldn't you be checking it too? damn straight! This is exactly why the OSS model is so damn important,

and its exactly why Microcrap, SCO, etc will never "get" it. I'd even add Intel to this list, because I think AMD is really "getting" it.

summary - we like it, we care about it, and aint no way we gonna let some dork attempt to ruin something we've worked so damn hard to build, not just for ourselves, but for everyone, its a matter of pride.

and yes, anyone found out (and they will be!) doing this shit is gonna get their ass kicked into next week...

In other news..

Microsoft insists the timing of their bounty (pay deal) on (for) virus writers (hackers) [slashdot.org] "entirely coincidental" (damned convenient)

Re:Why on God's earth...

You're missing something. They use bitkeeper, the CVS repository for people without CVS.

Its separate so they can screen CVS commits carefully.

Must Read: Ken Thompsom's Turing Award Lecture

Anyone that thinks security is easy (apparently some people still do) really needs to read Ken Thompson's 1984 Turing Award Lecture "Reflections on Trusting Trust": http://www.acm.org/classics/sep95/ As Bruce Schneier says, security is a process, not a product.

And this, dear reader

Should end the old argument about which is better, the habitual, easy to read, but easy to screw up or abuse:

if(variable == CONSTANT) { }

Or the safe version that's so much harder to screw up and which turns out to be just as easy to read with practice:

if(CONSTANT == variable) { }

Do we all understand the real world significance of this now?

If you still want to advocate (variable == CONSTANT), then please feel free to prove that no accidental or abusive (variable = CONSTANT) exist in the kernel.

Re:My boss is gonna read this..

No more Linux for us

Yeah, because he'd rather like a closed source product where such attempts suceed unnoticed.

Re:No one is mentioning this

Insightful my ass. Nowhere does it say CVS was exploited.

The code was injected into a CVS tree, the box could have been compromised in another fashion, such as a wu-ftp hole or some such thing.

So please, don't throw the word exploit around as if you have 1/2 a clue about security. It just makes you look silly to those of us who do.