More Info on Debian.org Security Breach

mbanck writes "James Troup (part of the Debian System administration team) has published more information on the recent compromise of four debian.org machines. The attack vector seemed to be a sniffed password of an unprivileged account, from which the attacker somehow managed to gain root and install the suckit rootkit and crack the other machines. As the machines were fairly uptodate with respect to security, an as-of-yet unknown local root exploit might be in the wild, so keep an eye on your boxen.Note that the main ftp archive running on a sparc machine was not compromised, so the exploit might not yet be ported to non-i386 architectures."

Boxen..

Here come the comments about the word "boxen..."

Re:Boxen..

Someone needs their ears boxen.

Re:So much for unbiased Slashdot

Look at all the posts...excuses and rationalizations. "Well, this serves as an example of weak passwords" or "non-root privileges," etc.

Actually, what I see is people warning of a possible security hole in the wild.

You never see that level of rational explanation when it comes to a user-transmitted e-mail Outlook worm. In fact, in those cases it magically becomes a "Microsoft hole," even though it's users running the executable!

This is because one of the "strong" points which is claimed by windows is that it's designed to be used by non-tech experts, while at the same time it offers NO protection from mistakes. If outlook were modified so that it cannot execute anything and you must manually save to disk and execute whatever you would see (beside a drop in virus infections) fingers pointed at the users instead of Microsoft.

Re:So much for unbiased Slashdot

Slashdot is NOT supposed to be unbiased. It's called /. for heaven's sake - if it was a Microsoft oriented site it would be \. (backslashdot.org)

Re:So much for unbiased Slashdot

Slashdotters are hypocrites and hold double-standards.

You're saying slashdot posters are inconsistant, but they're just different people who all happen to read slashdot. If you want to make a real argument, pick one person and attack their inconsistancies.

Another example is the political parties. You can't say that Democrats are inconsistant because of this, that, and the other. Democrats are a varied group, and they have many different perspectives and form their arguments in different, often contradictary ways. They just see a common means to their end, and each individual may be 100% consistant. (note: I'm not a democrat, I just used them as an example. This works with any political party that I can think of.)

Ultimately what you're doing is grouping variety of people together (slashdot readers) and then attacking the group as a whole for being inconsistant with respect to a separate issue (their perspectives about computer security).

You can do that to anyone. For example: "Blondes are so inconsistant. First they complain that the environment is being damaged, then the next week they're complaining about too much government regulation." Well, being blonde obviously has nothing to do with the topic, so of course you find inconsistancies in their viewpoint.

That type of reasoning is very simple-minded. The world is a complicated place with myriad possible groupings of people. Analogies that relate nations, corporations, SIGs, etc. to people often confuse the issue beyond repair. Microsoft isn't a "bully," it's just that the shareholders elect people that are likely to use aggressive business tactics and leverage the monopoly that they have to gain shareholder value. You can't punish MS in any way analogous to punishing a bully, because the shareholders could be long gone by now (however many years it takes to settle an antitrust lawsuit), because it's simply not a person, it's a group. Same with nations, it's a group and should not be personified. Think how much time the media has wasted talking about Bush as though he "doesn't play well with others." Nations are groups, not people.

Re:Boxen..

If you call your computers "boxen", I hope they get cracked and rootkitted.

Re:Boxen..

It's a perfectly good middle-english plural. Perhaps they just have rather olde boxen to develop on?

Two useful utilities to flush out the rootkits

Here are two useful utilities to flush out the SucKIT rootkit:


Kernel Security Therapy Anti-Trolls [freshmeat.net]

and

Kernel Security Checker [freshmeat.net]

Have a nice day !

Human Error

This incident reminds us of the importance of password security. It is sad to see one weak password responsible for such a breach. I think that it would be a good idea for the future to move away from the traditional unix password. An appropriate replacement would be something similar to RSA passphrase mechanism used by secure shell. A random passphrase with a minimum lenght would be idea. The user is the greatest security hole.

Re:Human Error

Random passphrase?

Repeat after me: The best password is the one that isn't stikie'd to the monitor and/or keyboard.

Re: Human Error

> Random passphrase? Repeat after me: The best password is the one that isn't stikie'd to the monitor and/or keyboard.

When it comes to internet-based attacks, my yellow stickies are the securest files on my system!

Re:Human Error

For some bizzare reason, I haven't found it necessary to be able to do that. All you need to do is learn how to make hard-to-guess, easy-to-remember passwords:
Choose a quote or sentence, take the first (or second if you really want it to be hard) letter of each word, use numbers instead of letters for words like 'to', and alternate capitalization for the rest:

"To be or not to be, that is the question" becomes
"2bOn2BtItQ" which should defeat any dictionary based attacks, and is incredibly easy to remember. Of course I also choose somewhat more obscure quotes or make up an interesting sentence.

-Chris

Re:Human Error

I say we cut out the user.

Re:Human Error

Clearly we need some way to move away from traditional passwords, but RSA keys isn't the way to go. They're impossible to remember, which means you need to store them on a computer. That makes them vulnerable to copying. You can password-protect them, of course, but then you're in the same situation as before (actually worse, for the same reason /etc/passwd is less secure than /etc/shadow).

That's not to say that RSA or some similar system won't be part of a good solution... but there definitely needs to be some other component. (For example, the private key might be encrypted by a biometric signature or keycard or similar. While that still leaves the system vulnerable to physical attacks, it more or less eliminates network-based ones as long as you use secure protocols.)

Re:Human Error

(For example, the private key might be encrypted by a biometric signature or keycard or similar.

I have yet to see a biometric signature that would solve this problem. Generally speaking, in biometric identification, information about the fingerprint/retina is stored on the disk and then compared against the data that is read in. The biometric information is not used *AS* the encryption key. So a biometric signature is just like a really big password, except that if someone cracks your password you can change it, but you can't (easily) change your fingerprints.

-a

Re:biometrics

Palm scanning only proves you have the hand of someone allowed to access a system. Retina scanning only proves you have the eyeball of someone allowed to access a system.

Well, the manufacturers of palm/retina scanners generally do include a feature that detects if the bodypart being scanned has a pulse. So you can't fool these scanners just by cutting off someone's hand or ripping out their eyeball. (Although it might be possible to manufacture fake contact lenses or glue-on fingerprints that would work.)

On the other hand, the basic weakness is that the biometric signature is still just a big password. You can "sniff" the signature by installing a fake reader. You can steal the signature off the harddrive of the domain controller. You can bypass the reader by splicing the wire. And your "password" is the same for every site.

Bottom line: I would sooner trust a token card.

-a

Re:Human Error

The biometric information is not used *AS* the encryption key.

And there's a good reason for that: It wouldn't work. Every time a biometric is scanned, the result is different. Biometric matching is hard because it's a process of evaluating the "closeness" of the livescan to the stored template and then deciding whether the two are close enough to be considered the same.

This means that trying to extract a set of bits from the scan which you could be sure would be the same every time is very difficult, and likely wouldn't net you many bits to use as a key. A set of bits that changes a little every time doesn't make a useful key.

Given some sort of a secure processor, you can store the key and the biometric template in there, and program it to refuse to use the key until it has been presented with a biometric scan which it considers to be close enough to the template. That gets you about half way to security, now you just need to find a way for the secure processor to verify that the livescan it receives is fresh, and not replayed. Oh, and it would be good if you could also be sure the livescan is a *live* scan. And don't forget to secure that template database well.

Making biometrics secure is hard. In practice, this means biometrics are only useful in two situations. The first is very low security, where the biometric is being used to raise the level of security from very, very low to very low. The second is very high security, where the biometric is to augment some other authentication methods, or when verification is only done in a very controlled environment, i.e. where you're watched closely by a human guard who knows how to ensure you're not trying to fool the scanner.

Re:Human Error

Er, the problem with biometric identification is that (1) its not testing who you are, just that the digital input matches some value and (2) you can't change what its testing.

You can't change who you are. Thus, once the key is compromised, it stays compromised.

Re:Human Error

A good method: Easy mental ciphers.

You pick a passphrase that you use for all of your systems. You then pick a unique seed for each system. Then, you do some quick mental math on it (pick an algo of your choice, just make it simple) and then you have the effective security of two passwords + unknown algorithm. It will make all of your passwords invulnerable to dictionary attacks (unless a rare circumstance has your resulting password being "password" or something)

For example, if you have a pass phrase of "MYBOXISSECURE" then you can use the box name as a seed, lets call the box "DEBIAN" and have the algorithm block the seed and then subtract, modulo 26.

MYBOXISSECURE
DEBIANDEBIAND
-------------
I'm too tired to do this and I'm on my windows sytem without perl.

Then reverse it or something. Walla! Pseudo-random passwords. Works great, and after a few times you will memorize the keystrokes and you won't need to do it by hand. You can even have a standard system for the passphrases amongst an entire group for the root password, so each system can have a different root password that everybody can just figure out as long as they know the passphrase. In addition, if you want to remove someone from the loop, just change the passphrases and redistribute to the trusted source.

It's a hack solution for the weak-password problem.

Re:Human Error

Uhh, I dunno if you noticed, but it wasn't a password alone that did this much damage. The account broken into was unprivellaged, meaning it was just a simple user account.

In theory, a secured system can have this happen to it and the attacker will have fun deleting a single home directory before they run out of damage to do.

In practice, a single local privelage escalation attack is all it takes. Maybe this will end up being a good thing in the end, we get to find a previously unknown local root exploit, fix it and improve the Debian security practices, all in one move.

Re:Human Error

So when an exploit is found in Windows, it is considered a bad thing that shows how lame of an OS it is.. but when it is found (or not?) in Linux it is a good thing?

Yes. In the past, Windows exploits get found one of two ways. The first way is when a virus is found in the wild. The virus is deconstructed, then Microsoft does a cost analysis to determine if it's worth patching the vulnerability that enables the virus. If so, then a binary only patch will be issued. The first you'll hear of it is when you're able to download the patch. The second way is when a white hat hacker or security analysis team at some college find an exploit. If they go public with it, they're criticised for not giving time for Microsoft to develop a patch. If they go to Microsoft with it first, then the cost analysis process starts, only because the public at large doesn't know a problem exists, there's a much smaller chance a patch will be issued. In either case, the patch may or may not work, and it may or may not break your system. Caveat emptor.

When an exploit is found in Linux, it gets fixed. The cause of the exploit gets scrutinized world over, and other developers privately consider whether their software might have the capacity to be exploited in the same way.

Password was *sniffed*

Apparently the password was sniffed [google.com]. This generally implies that it was obtained through monitoring network traffic and seeing it trasmitted in cleartext. A strong password wouldn't help here; only a good protocol would.

This was both user and admin stupidity I guess. Admins who care about security shouldn't permit access through cleartext passwords and users shouldn't send their password in cleartext if they care about their account. Unfortunately many users don't know about this risk.

Re:Password was *sniffed*

Why assume it was a cleartext password? It could of been encrypted, captured and crack via brute force or some other method.

Re:Password was *sniffed*

Ah, but the SucKIT rootkit is particularly useful as it captures all tty i/o at the kernel level: all interaction with sshd is captured in a "sniffer" file. No decryption or packet sniffing needed- the attacker owns the system completely if they have installed SucKIT. If you don't trust a computer that you have ssh'd into, never ssh or scp from the untrusted computer back into your trusted systems. If the untrusted computer has been compromised, any login sessions that you have from the untrusted computer will expose the passwords if a SucKIT rootkit has been installed.

Human Error or faulty security models?

SELinux would likely have prevented the root exploit from allowing this individual from doing as much harm as was done.

I think that it's time for the big names like Debian, Slackware, Red Hat etc to start implementing it on their network connected machines. It's being incorporated into the stock kernel for a reason. Use it!

Unknown Debian exploit?

Im sure glad my network runs on Windows!

Re:Unknown Debian exploit?

Im sure glad my network runs on Windows!

hey it is pretty nice - i'm having a look around right now!

In a nutshell - somehow

Quote from the article:

"Somehow they got root on klecker and installed
suckit."

What follows is an interesting read - but the guts are in that 'somehow'.

Re:In a nutshell - somehow

You're absolutely right. For some reason, everyone else seems to be overlooking the fact that there is (or appears to be) an unknown root exploit out there.

Yes, you can probably guess/crack/social engineer a password if you try hard enough. That's why security is about layers, compartmentalisation and multiple types of protection, not just a single password.

If this was your box, would you be more worried that someone had managed to sniff an (unprivileged) password? Or that any one of your users can now root your box? I know which one I would lose sleep over.

Here's to hoping that the root exploit is found and patched nice and quick. Even better if it something else that's been missed and is fixed in the latest patch.

Re:In a nutshell - somehow

For some reason, everyone else seems to be overlooking the fact that there is (or appears to be) an unknown root exploit out there.

Uhm, did you read James' post? Here's a quote:

Unfortunately due to the fact there is (I believe) an unknown local root exploit in the wild, we can't yet unlock the Debian accounts.

Surely this constitues something else than "overlooking" the root exploit? Deciding to keep the Debian accounts disables effectively stops the entire developement of Debian. Nobody has been able to upload packages in the last week, and lots of services are down.

James could have unlocked the accounts to make the developement pick up again rapidly (which would probably would be the only option in a corporate setting -- there's a release schedule that must be kept at all costs), but the admins are being thorough on this one.

In summary: James (and the other admins) are keeping the entire Debian Project in suspense for the purpose of tracking down this local root compromise and preventing it from being exploited again. You might want to think about that for a second, and see if "overlooking [the] unknown root exploit" is applicable here.

Diebold, take note

All vendors and site administrators should take note of the openness with which the problem was dealt.

When I go to buy a car, a computer, or a stereo, and the saleslizard is cagey about any problems that come up, my trust level goes down. If they tell me all about all the problems with the thing they're selling before I even notice them, my trust level goes up. It's like a cool drink on a hot summer day.

Contrasting with Debian, how long did it take to find out that Diebold ATMs had been hit by the Nachi worm?

I'm now more inclined to trust Debian, and less inclined to trust Diebold.

Re:Diebold, take note

More importantly, the openness of Debian is a much more important factor here. When I read these lines in the article:
The attack vector seemed to be a sniffed password of an unprivileged account, from which the attacker somehow managed to gain root and install the suckit rootkit and crack the other machines. As the machines were fairly uptodate with respect to security, an as-of-yet unknown local root exploit might be in the wild, so keep an eye on your boxen.
I got the distinct impression that Slashdot is transformig into a FUD channel for unsuspecting readers.

The fact that a 'clean' Linux system can be backed up and restored from any media, is of more relevance and importance to users. EVERY system connected to the internet has potential unknown vulns, those running Windows are often unpatched and have no disaster control system as well.

Viewed from this perspective, I don't think we need to keep an eye on our boxen just the backup tapes / disks/ CDs.

-

This attack has obviosly shocked the comunity.

Since Debian (even for those smart ones out there using slackware, like i do) is really considered one of the real distros, if we hear that redhat has been atacked, we would just say that they diserve it and go on, it would be delivered in the respective mail list, and that was it.
But this attack has a psicological impact. Debian itself has been attacked, and it seems to be a bug exploited just in part, on the other side, there are updates that the compromised machines never got aplied, and other big mistakes like a non-tared backup lying arround, with the original owner / permissions mask. This is really more that enough to get any netadmin running Debian to get paranoid.

One recommendation

Off-site logging of all accesses.

One of the first things that get wiped in an intrusion are the logs. All access logs should be copied in as near real-time as possible to a remote server that is not accessible from the machine being logged, i.e. a drop-box.

Re:One recommendation

One of the first things that get wiped in an intrusion are the logs

Try wiping logs printed out on a matrix printer...

Re:One recommendation

With physical access, all bets are off at the best of times.

Printing logs is a good idea in some circumstances; you will have a record of all actions and a remote intruder has no method of editing those logs. The main downside is the amount of paper it could use, plus it has to be kept supplied with paper & ink.

Re:One recommendation

To make sure my logs are secure, they are automaticly:

• posted to several usenetgroups
• posted as random comments to /.-stories (Along with some random anti-SCO/Microsoft propaganda so I don't get modded down and don't lose karma :)
• uploaded to the linux kernel CVS
• sent as email to all my friends

This way they are mirrored as many places as possible and hopefully cached by Google. Wipe that out!

Great

Right as I am downloading Debian.
I will check the md5sum.

Anyways Something to be said about passwords.. I am getting sick of passwords.. I have looked at the RSA keychains, But they cost too much.

So I ask are there any good one time password systems out there. That are opensource.. I have looked at going with smart cards but again with the money. (not to mention overkill for me)

I have found a few but none with a keychain.. I don't mind paying for a keychain, but I want the software to be opensource.

Re:Great

Probably the closest you'll get to a "good" system would be something like S/Key or Opie (debian packages: opie-server, opie-client, libpam-opie - Use OTP's for PAM authentication) for generating and using a one-time-pad of password systems. The issue in this is that you must generate the pad in some secure fashion, if someone sniffs your pad because you downloaded it over the network, you've lost.

You could easily keep a pre-generated giant pad itself on a usb drive or something similar.

Root password

Once an infiltrator is in a machine, it is often just a matter of time before he acquires root access - unless monitoring or disablement are standard procedure.

Depending on the power of the box and the time from which the lower-level account was compromized, it could just be that a password-cracking procedure gained root access. Of course, it's also possible that the attacker managed to nab control of a process running as root, but again the initial compromise still required cracking a password to gain access to the machine.

First rule, secure your passwords... and it's probably not a bad idea to use a password cracklib to ensure that any semi-privileged (can SSH) users have somewhat secure passwords as well.

Proof that Windows is more secure

Not really, just thought it needed to be said.

Easy solution

Install windows. You'll never have to wonder if your system is being compromised, you'll know it is.

Oh, and "password" is not really a "password".

#1 on Ten Immutable Laws of Security

I worked at Microsoft, so Microsoft's list [microsoft.com] is my frame of reference:
Law #1: If a bad guy can persuade you to run his program on your computer, it's not your computer anymore.

Ammended for the rest of us:

Law #1: If Bill can persuade you to run his program on your computer, it's not your computer anymore.

Re:#1 on Ten Immutable Laws of Security

Law #1: If a bad guy can persuade you to run his program on your computer, it's not your computer anymore.

That's why I've been saying for years that all my computers are owned by Bill Gates.

Re:#1 on Ten Immutable Laws of Security

Not that I even like Microsoft's security list, since it's very Windows-centric, I'll bite.

Law #1 doesn't apply here. The intruder sniffed a password, and ran his own software. As far as I know, nobody was tricked into running malicious software. Law #1 should read, for real OS's
"Law #1: If a bad guy can persuade you to run his program on your account, its not your account anymore."

The first failure, as per this list was Law #5 "Weak passwords trump strong security." Someone didn't properly protect their password, this gave the attacker their foot in the door.

The second failure was the unidentified privilege escalation. This doesn't appear to fit any of the laws (they appear to be written assuming privilege escallation is trivial, I guess that says something about Windows). Except perhaps, Law #10: "Technology is not a panacaea". Just because we run well designed software that has few security holes doesn't mean that we run perfectly designed software that has no security holes.

Occasionally something slips through the cracks, like here, and it's good to know that real people are paying real attention, and that there are effective ways of bringing necessary systems back up in a trusted fashion. Eventually, this escallation will be found, fixed, and machines patched.

A simple disaster-mgmnt starrtegy...

Since Linux has no use for hidden files, registry, active directory, complicated booting procecdures and other useless features that come standard with Windows - I see no point getting worked up about these so-called Security Warnings.

99% of Slashdot readers, I believe, treat viruses, worms and other 'security' attacks as a NUISANCE rather than a PRIVACY hazard. A Service Pack or bug fix a week for Windows merely highlights the fact that data privacy on a 'personal' computer is a joke. The nuisance of reinstalling the Windows OS from CD, and reinstalling each and every app with the zillions of settings OR buying expensive, uunreliable 3rd party s/w for disaster recovery can be intolerable.

With Linux, OTOH, simple tools exist that can take backups of disk data (not disk images, just the files), AFTRER installing the apps. A simple restore of these files gets the system back, with all settings and screen-savers intact.

To sum up, 99% of Slashdot readers do not need to care about these security risks, if they choose Linux for their personal or office systems.Those with Windows - a switch to Linux is cheaper than anti-virus s/w PLUS OS cost PLUS frequent updates PLUS frequent reinstalls PLUS loss of data PLUS nuisance.

-

SELinux

SELinux would likely have prevented the root exploit from allowing this individual from doing as much harm as was done. I think that it's time for the big names like Debian, Slackware, Red Hat etc to start implementing it on their network connected machines. It's being incorporated into the stock kernel for a reason. Use it!

What could be done better...

Quote: "All the compromised machines were running recent kernels[1] and were
up-to-date with almost all security updates[2]."

Well, it seems that 'almost' just isn't good enough. Perhaps there is more to the break in (like unknown holes)?

Sniffing passwords? They must be using 'almost patched' version of SSHd.

Openness is good

I like how when debian's servers are cracked they tell you about it and furthermore, remind you again later with the details. If a similar thing happened with Microsoft it would be hushed down and certainly no details about it would be publicized later. Come to think of it, even a commercial Linux company like Red Hat might be weary in dealing with a similar issue as well -- I think they'd be likely to be open about it, but you never know what's going to happen when money and stock prices are involved.

oh, you didn't know

clearly this is the work of a DX fan. (wwf reference)

local root == remote root

This is a good demonstration that the distinction always made between local privilege-elevation bugs and remote exploits is academic hair-splitting. It's rarely difficult to get unprivileged access through a buggy non-privileged service. (Web-server plug-ins are a reliable source of entry points.) Once you're in, privilege elevation takes you the rest of the way.

Certainly the distinction is useful to security students and analysts, but it's misleading for everybody else. "Oh, that one's just a local exploit; not so bad." The OpenBSD advocates promote the fallacy: "only one remote exploit in this millennium!" (or something like that), encouraging us to ignore almost equally damaging exploits in non-core services that provide access to local accounts and more damaging attacks.

There's a similar fallacy in distinguishing security holes from other bugs. Without a depth of analysis that hardly anybody can ever afford, almost any bug might actually be a security hole, too. The OpenBSD people get this one right -- to them, any bug is a security hole until proven otherwise, and they encourage running latest versions -- but almost everybody else gets it wrong. When I fixed a double-free segfault in lib[mumble], nobody posted security warnings about every program that relies on it. despite that double-free bugs can often be exploited.

Debian gets this wrong, and very selectively backports only proven security holes, ignoring the myriad bugfixes that might just as easily be security holes as well. To find holes in stable-branch services, just look for bug fixes in later versions, particularly in libraries used by those services. Failing that, look at new features added shortly before the library-version used. Chances are the last new feature added has bugs that haven't been noted yet, and that might be exploitable.

This might be a good place to mention that the CVS codebase is almost irreparably insecure. The practical implications are: (1) A remotely-accessible CVS server should never be run on a host that does anything else that matters, or that has access to anything else; (2) An anonymous CVS server should never be the same CVS server that is used for checkins, or even run on the same machine. The pserver should be a slave that only gets read access to a copy of the archive. (3) Checkins on remotely-accessible servers should result in patches logged to another archive kept on another, not-remotely-accessible machine. Patches from that server should be posted to the mailing list.

local root != remote root

Huge diffrence.

You still need a local account to make use of a local root exploit.

You don't for remote root exploits.

Remote root exploits can be used in worms, local (for the most part) cannot.

Not to say that local root exploits should be overlooked, especially when they seem realtivly simple to create (e.g., bad symlinks)

Besides, this is supposedly an *UNKNOWN* local root exploit..

Of course there are unknown exploits

The serious attackers don't publicize the ones they develop. They save them for use on worthwhile targets.

This is why security by patching is fundamentally ineffective against enemies, as opposed to nusances.

Sad day for Debian

As long as a machine is connected to the internet there is going to be a method to compromise it. My question is this why Debian? They are the only Linux distribution that is truly built by volunteers to gain any mindshare of real note. (not sure about slack so please dont sick bob dobs on me) This is not imhop the work of rank amatuer crackers with there first root kit. These were servers being run by experienced admins using a distro known for stability which when patched and up to date usually means somewhat difficult to hack. I seriously doubt these guys were running winders attempting this either. Wtf is happening to the community when people with talent are attacking a distro that yet again imhop doesnt suck. These guys need to be found and buried. Not by the police but by the commmunity. Last but not least (places tinfoil hat on head) could this have been funded by M$ trying to discredit linux. I cant see the glory angle so its got to be money or power. (no glory in getting called a dick when you tell your friends what you did)

Re:Sad day for Debian

> Wtf is happening to the community when people with talent are attacking
> a distro that yet again imhop doesnt suck. These guys need to be found and
> buried. Not by the police but by the commmunity.

hear, hear.

it's not a sad day for Debian so much as it is for the community. if Debian can find this supposed new exploit, fix it and publish details, then Debian will rise a little higher in people's esteem.

but why crack Debian in the first place? here I am stumped, but then I've never fully understood the cracker mentality.

ldap?

Obviously we can't continue without LDAP accounts for very long either.

Can someone who's familiar with system administration on those debian boxes clarify the above statement? Have they disabled LDAP accounts or was it implied that they're going to set up authentication with a ldap backend in the future. If it's the latter then I'm curious as to how having ldap in the equation would have made cracking those system accounts harder.

SuckIt Exploit

I have dealt with this rootkit for nearly 4 months when it first appeared. The fairly safe methods for avoiding this is by 3 steps which I have used and it works well since then.

Move the /tmp to it own partition and set it as noexec, nosuid and give it plenty of space, around 200 to 500 megs for it.

Patch the kernel with either Grsecurity or Openwall Patch on 2.4.22 kernel and set it as mononthlic kernel, not modular with no open hooks for adding additional modules.

Then I installed the suphp module for PHP to run scripts as users instead of nobody, especially when people trying to exploit it. I get it at www.suphp.org and it works extremely well. Since the changes, I haven't seen any rootkits being successfully implemented on the servers I admin. And note the fact that I manages over 260 servers for various clients points to the track records.

What's up with these anti-Linux attacks?

To me, this attack and the recent attempt to insert an exploit into the Linux kernel [iu.edu] seem like possible evidence of a disturbing trend: skilled attacks against high-profile Linux sites (you can't get much higher-profile than kernel.org or debian.org). I'm pretty sure that these systems were secured against all known local root exploits; if they weren't, this probably would have happened long ago.

So, what's going on here? Are these simply two unrelated attacks? Is it an attempt by an immature highschooler with some cracking talent to boast to his friends "LOL 1 hax0rred debian.org!?" Is it an attempt by some sort of anti-Linux commandoes to undermine Linux's public image? I almost suspect the latter, but the prime suspect there is Microsoft, who have far too much to lose by going that route and plenty of money for traditional FUD that will make it into "traditional" news channels better anyway. SCO might be crazy enough to do it, but they probably wouldn't want to divert resources away from spewing lawsuits at everyone in existence.

From what I understand of the cracker community, Linux is held in fairly high regard (although I admit I don't try to keep up on the latest in the cracker community). You'd think that black-hats, who tend to be rather immature, when armed with a brand new exploit, would attack a site seen by the general public and post goatse.cx images on the front page, rather than subtly changing Debian packages. So, who's behind all this?

I just can't wait...

Until the next announcement comes out, further explaining the possible motives and more on how they exploited the system from an unprivileged account.

This is like a soap opera!! =)

P.S. I don't like soap operas, but if they had one about this, I might just have to quit my day job.

Debian physical site security?

Okay, I read the article and it said that at least one machine was at a remote location that couldn't be accessed - can anyone tell me what kind of physical setup debian project uses? I always get the impression that they're based out of some dude's dorm or basement, like in this OpenBSD image [openbsd.org]. Do they have any physical security measures at all around their boxes?

Re:Debian physical site security?

The primary Debian machines are in colo facilities
in the US and Netherlands (there are buildd machines available to debian developers in various locations). The machines are beefy enough - HP
recently donated a server with 48 GB RAM, for example. I believe the bandwidth out of ftp.debian.org is Gigabit ethernet (and having only that to the mirrors will be a bottleneck
when sarge is released!)

So, no, they're not in some dudes basement; we have good facilities courtesy of our sponsors.

- Alastair

Something big is missing

Somehow they got root on klecker and installed suckit.

"somehow they got root" isn't very informative.

the linux attitude prevents real security...

it's a sad thing that everyone seems to be so confident in their latest super secure linux setup, the power of fast and often patched open source software or the openess in such issues - so much that nobody takes these problems serious enough.

for every exploit known (and fixed) publically you can bet there are two yet undisclosed and maybe in the hands of the wrong people...

concepts like public key crypto (ssh, ssl), stack guarding (say no to buffer overflows) or process jail (try to escalate privileges from there) are thus essential to implement real security. still ease of setup or performance seems to be more important than safe networking.

perhaps the big desaster has to happen before people understand that projects like openbsd or selinux are not your tinfoil-hat wearing neighbor's business but the only serious choice for any public, responsible service provider.

suckit ...

This reminds me of a shit we had back in the april at the place where i work. We got a couple of production server r00ted with suckit, with the only possible attack vector being apache/php (only port 80 was open in the firewall), that were latest versions back then. The only way to stop it was to recompile a kernel without modules support and some minor patches to deny writes to /dev/kmem in any possible way ... therefore killing the method suckit uses to load itself. See point 6 here [phrack.org] and here [epita.fr].
There were quite a lot of similiar reports from the folks all aronud at that time ...

My big hairy conspiracy theory would be in the line of super zonda type of organization hiring some of the most skilled crackers and r00ting the boxen all around ... for spamming, ddosing or whatever ... welcome to the Wild Wild Net.

Doesn't seem very logical..

An attacker who has access to unpublic local root exploits probably won't use a public kiddie-rootkit like Suckit.

And I hardly believe that an experienced cracker would backdoor the boxes in such an uncareful manner. Weird..

Why might ssh keys be compromised ?

I've a couple of questions on this.
There was a comment in the origanal posting that remote SSH keys may be compromised if authentication forwarding was used ? Can anyone explain why that might be ?

Also, isn't anyone worried that the only reason the root kit was discovered is because *two* machines were oopsing. ?

Defeat binary evil!

Note that the main ftp archive running on a sparc machine was not compromised, so the exploit might not yet be ported to non-i386 architectures.

So if we run Linux on Sparc, and Solaris on x86, we're safe!

No OS can ever be 100% secure, but.........

Proprietary OSes will ultimately be left behind Open Source OSes in terms of security for the following reason. In the fight against proprietary OS's such as Microshaft's, there is a big propaganda war with both sides saying "Look, your OS is insecure". Both OS's will have security holes discovered, and hopefully fixed, from time to time. That is a fact we have to live with. The rate at which they are discovered and fixed is roughly proportional to the number of people actively investigating holes in the OS (ignoring the fact that there might be other, political reasons to look for security holes one OS rather than another). However as time goes on, we should expect the number of users of Debian (and GNU/Linux in general) to increase, hence the number of people discovering and fixing security holes will go up in proportion. This is the 'many eyeballs' effect. this will lead to GNU/Linux becoming ultimately very secure. In contrast the number of people actively looking for security holes in, say windows, is proportional to the amount of money their perpetraitors (sic) are willing to spend in this task. This does not go up in proportion to the number of users. In fact as competition pushes prices down for proprietary offerings, the perpetrators find they have progressively _less_ money to spend on looking for security holes. Ultimately they will get left behind. So we should see that Open Source OSes such as GNU/Linux will become more and more secure at a rate which accelerates much faster than for proprietary OSes. At the moment, we have one OS which is used by 95% of the world's desktops, and scores fairly low on security (although it is improving). On the other hand, we have GNU/Linux which is used on something like 2% of the world's desktops (more on servers), and scores fairly high on security (although it's not perfect). So from this small user-base, we have already benefitted from the 'many eyeballs' effect of Open Source to gain an advantage over the competition in this respect. This advantage can only accelerate, for the reasons I have outlined above. Ultimately we should expect to see Open Source winning on all fronts in terms of reliability, functionality and security. It will never be perfect and there will always be crackers trying to spoil the party, but it will be a lot better than today's situation. We just need to work hard to make this happen sooner rather than later, as it will be a long haul...........

Other distros affected???

Everybody here is talking about an unknown exploit in Debian. What I haven't seen is a discussion on the probability that this might affect other distros too. Is it Debian specific, or Linux, or even UNIX (based on an app) specific? Let's not be complacent here.

How much would writing applications in Java help?

I know slashdot is almost a java free zone but:

With java
-There are mo buffer over runs or memory allocation issues, but then
-There may be bugs in the java implementation. -Cleartext passwords would still be a problem if the app is not written with security in mind.

Deep problems do not go away, but java solves/automates the solution to many shallow problems.

the unknown

This is really the heart of the issue: the unknown exploits. I've often been at the forefront of theorizing about possible vectors for unknown exploits. I'm usually flamed severely for it. The fact of the matter is that these unknown exploits exist and people need to be ready to deal with them.

If a "bad" hacker comes up with a new root exploit he's not going to e-mail all of the "good" hackers and let them know. He's going to make use of it mercilessly until he's noticed and caught. Microsoft ignores this issue outright and the OSS community tends to skate around it. If the computing public as a whole knew the facts about security then McAfee and Norton wouldn't even be in business. "Updating virus definitions" twice a week is still going to be ten weeks behind the hardcore caffeinated malicious hacker.

The OSS community has dealt with this issue in the most productive manner possible: complete openness and timely notice. Microsoft, on the other hand, would happily allow millions of users to remain compromised for months or years until their internal programmers manage to find the "unknown local root exploit". This could easily result in identities and credit card numbers stolen, bank accounts infiltrated, and possibly even malicious interference with real life relationships and employers just for fun.

Should the software manufacturer be liable? No. Should the user be entitled to know? Yes.

The OSS community is the only solution which addresses this situation correctly.

time of attack

take a look at the time of the attacks.

The attacker stoped at 19:00h and started back at 5:00h.

Now from this time, and assuming the attacker is like most computer guys I know, he would sleep around 2-3 AM, and wake up around 11-12.
This would place the attacker at 6-8+ GMT hours.

that is, in china, Mongolia, Rusia or Indonesia.

7 degrees of seperation

That principle suggests to me that no system into which ssh (or indeed any secure) access is possible is safe.

Here's why

You go in through the easiest system you can that has people that use the target or use something that is a step on the way to the target.

Once you have the ability to use suckit or other ways to compromise SSH on any machine you hace access to the next machine along.

Eventually if you are quiet and patient you will get somewhere interesting.

So who is secure?

What debian's not said, clarifications speculation

I beleive the additional details of this exploit are roughly:

A debian developer (who I'm not going to name but it's not exactly a secret) revealed his password by logging into some machine that had been rooted. Shame on him for using the same password, and the Debian project for not policing that kind of thing. (That said, people do this all the time, even people who do/ought to know better.)

The password 'sniffing' being referenced is not sniffing network packets but rather session IO. If you read the 'developer cleanup' instructions it will be clear that they beleive that the 4 dev boxes that were rooted were being used to collect account and password info from developer's sessions. (Another procedure error, the systems in question probably should not be allowing users with shell access to ssh out to other machines.)

There has been a LOT of speculation that there's a privilege-escalation vulnerability in the kernel version running on the target systems and/or up to the 2.4.22 kernel (I'm dubious, however 2.4.23 has just been released today so who knows).

As many here and elsewhere have wondered, it seems unlikely that a 'kiddie would have access to somthing not yet observed in the wild, and if this is the work of more capable 'bad guys' then it seems equally unlikely that they would have been so noisy as to have been caught in less than a day.

Leaving us really not knowing much about the state of either debian or the kernel at this time. I certainly hope that a more complete complete 'explantaion' will be coming, hopefully soon.

We had similar experiences recently

Our Linux firewall was compromised a while ago. The attacker apparently used a samba exploit to break in (RedHat 9.0 with patches installed) and then tried a bunch of sendmail, NFS and samba exploits on other machines. Its not clear how he got root access but it seems he replaced some libraries with a few extra routines. After becoming root, he replaced all files in /bin and tried many exploits on the win2000 servers.

We were lucky not to have been hit hard, but the scare made us replace the Linux with an OpenBSD firewall with tightly configured ipf and snort. In the proceeding days, we logged attempted telnets and ssh from 3 different IPs 2 in korean primary schools, 1 from a chinese telecommunication company. (APNIC)

So the RedHat has been moved to be an internal server and we're careful about packet filtering now. Theres a finer level of permissions control all over the place and all services run as nonroot users, some in jails.

What excellent FreeBSD security(7) says

One must always assume that once an attacker has access to a user account, the attacker can break root.

So nothing special about that break-in. In long time run such incidents must happen on any machine with publicly available accounts.

Exploit might not be debian-specific

Our server has been hacked twice in a 3 weeks time :

at first we had a redhat70 fully patched and up to date. we never managed to find how :
1) he got in
2) he got root

he installed 3/4 rootkits and defaced all our html with a javascript/iframe pointing to http://viprating.com/rate and http://viprating.com/counter
there, tripwire saved my life!
neverless, we reinstalled every thing with a debian woody.

3 weeks later (4 days ago), the (same?) hacker broke in using :
1) an apache 1.3.26 shellcode which attempted to install linux.JAC virus from http://www.infosmolensk.ru/c [217.107.188.155] (beware! virus!), apparently failed to run
The logs showed some shellcodes, followed by a wget
2) second apache 1.3.26 shellcode followed by an unknown root exploit
The logs showed some shellcodes, but no command output
Then he installed suckit rootkit, and defaced all our html with a javascript/iframe pointing to http://viprating.com/rate and http://viprating.com/counter

I found the attack came from a russian site, http://www.infosmolensk.ru [217.107.188.155], as a saw several established connections from this IP to our port 80 with apache stopped...

Yesterday, he tried once again, but hopefully apache 1.3.29 behaved better :

access_log: 217.107.188.155 - - [27/Nov/2003:08:00:02 +0100] "\x93OZ\xe0\xe6\xe6\xef\xaeDRRq\x8dg\x82\xe1\xfd\x fe\x84f\x93OZ\xe0\xe6\xe6\xef\xaeDRRq\x8dg\x82\xe1 \xfd\xfe\x84f\x93OZ\xe0\xe6\xe6\xef\xaeDRRq\x8dg\x 82\xe1\xfd\xfe\x84f" 501 - "-" "-"
error_log: [Thu Nov 27 08:00:02 2003] [error] [client 217.107.188.155] Invalid method in request \x93OZ\xe0\xe6\xe6\xef\xaeDRRq\x8dg\x82\xe1\xfd\xf e\x84f\x93OZ\xe0\xe6\xe6\xef\xaeDRRq\x8dg\x82\xe1\ xfd\xfe\x84f\x93OZ\xe0\xe6\xe6\xef\xaeDRRq\x8dg\x8 2\xe1\xfd\xfe\x84f

I hope this exploit will be found 'cause we updated apache and put a kernel without LKM, but could not find the exploit!!!
Maybe the same exploit was used against the redhat... which means it might not be debian-specific!
Once sure thing : it is the same hacker, cause the defacing was the same

Re:But wait....

When has windowsupdate ever been compromised?

Windows Update is served by Akamai ie Linux.
Now what was your point ?

Re:at which point

So that means Monica Lewinski wrote the Suckit rootkit?

Re:Ask Slashdot

Soviet Russia, 6 5 4 3 2 1

Re:Security "experts"

To be fair, not even the technical consultant types and the security "experts" get it right all the time. The real world is very, very different from the edges of theory.

Furthermore, just because someone has "at most" a degree in English doesn't mean fresh insight hasn't been exposed. Narrow-minded elitism gets us nowhere.

Re:Nmap says enough about the quality of debian !!

That tells you they are running an ftp, ssh, rsync and a web server - hardly surprising.

'Filtered' means that the port is blocked by the firewall, as opposed to closed ports which can be reached through the firewall but have nothing listening on it.