Debian Project Servers Compromised

Sean was one of many to pass along the bad news from the debian-announce mailing list: "Some Debian Project machines have been compromised. This is a very unfortunate incident to report about. Some Debian servers were found to have been compromised in the last 24 hours. The archive is not affected by this compromise! In particular the following machines have been affected: 'master' (Bug Tracking System), 'murphy' (mailing lists), 'gluck' (web, cvs), 'klecker' (security, non-us, web search, www-master). Some of these services are currently not available as the machines undergo close inspection. Some services have been moved to other machines (www.debian.org for example). The security archive will be verified from trusted sources before it will become available again." They were going to announce 3.0r2 this morning; they've checked it and it's unaffected but obviously they're still postponing that release.

Grumble, grumble

What's interesting about your comment is that when a M$ compromise comes to light, the focus is on how big a bozo BillyG is for letting his insecure crap out into the world. When something like this happens, its those nasty little hackers or script kiddies and their deep dark motives or a cabal led by M$/SCO to "discredit" Linux. Face it, the main servers for a major distro was hacked into at a very sensitive time. Ouch. Regardless of the whys of who did it, it was done. Yeah, kudos for them coming public, but if I joe CTO and looking at purchasing some puters, I'm thinking to myself, hey, what's up with this, they told me that M$ stuff sucked and this Linux stuff was secure. This wasn't some ma and pa website that got defaced after all.

password

You know what... encrypt your SSH connection at 1024-bit... lock your webserver in a vault, 2km underground, with triple combinations... post armed guards... lock down all ports except port 80 and SSH/whatever.

Then, have your password stolen, and oh shit, you're compromised. It's not about the OS being insecure, it's about a lost password. NOTHING can protect against this, short of one instance I heard where updates required 3 user passwords (from 3 users), but what a pain that would be.

Re:...not the archive.

How does this change the fact that Debian is just not good enough, and has compromised thousands of machines across the globe? Sheesh, the denial... This is just like the Mandrake frying standard PC hardware story.

As far as I understand, no machines apart from the several Debian computers have been compromised. Compromising a machine that hosts the central Debian APT repositories is a perfect opportunity for backdooring thousands of machines In this case, that didn't happen. "Thousands of machines across the globe" have not been compromised. I guess it's only good luck but Debian users were not affected by this security breach.

Re:...not the archive.

The server that pushes .debs to archive is running debian/sparc (donated by sun btw), so probably the cracker didn't know how to port his leet exploit to sparc (all the comprimised machines were 1386).

Not on debian-announce archive

The debian-announce archive [ http://lists.debian.org/debian-announce/debian-ann ounce-2003/threads.html ] doesn't list this message. Of course with the number of machines affected it's possible that the mailing list archive is somehow affected.

-JohnF

Re:Not on debian-announce archive

Yes, lists.debian.org runs on one of the compromised machines and is, er, not quite running on all cylinders just at the moment.

Re:Not on debian-announce archive

As other readers have pointed out, that machine was apparently affected.

I got the email too, and I checked its Received: headers against a debian-announce message in my mail archives from about a year ago. They both came from the same source. So there's no way this is a hoax ...unless the murphy.debian.org machine that emailed it to me is compromised, in which case it's not an inaccurate hoax :/

Re:Where's the confirmation from debian people?

At least cjwatson and myself are Debian developers. I wish I could say it's a hoax, but it's not. However, as you've already read: the archive doesn't seem to be compromised at all.

Re:Where's the confirmation from debian people?

--- snip here ---
This is a truthful report.

You may validate this message against the key for [e-mail].

Steve
--
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.2.3 (GNU/Linux)

owGbwMvMwCR44PyxzWd9eOcyns5PYrDfJ7EiJCOzWAGIEhVK ik pLMtJKcxSKUgvy
i0r0uLgi80sVchMrFcoSczJTEktSFUpAin NTi4sT01MVEtMTM/ OKS4CCqQrZqZUK
aflFCsXZFQ4pqUmZiXl6+UXpQCO4gktSy1 K5dHW5OuyZWUE27o M5QZDp9w6GBQtO
SLxI+1madnvjbIZVrZu0HcTnzGdY0LBFy+ hFp+fRBXM7HXcYc1 6Xj5A9DwA=
=xVtr
-----END PGP MESSAGE-----

Re:Where's the confirmation from debian people?

Thanks for confirming this. Unfortunately, the way you confirmed it is very dangerous.

Your message contains:

• no date
• no precise reference to the report that you are confirming

So from now one, your "confirmation" can be used by anybody who wants to claim that some random report of theirs is "confirmed by a debian developer". Until you revoke your own key, of course. That's a pity.

Re:Where's the confirmation from debian people?

Not to be pedantic, but the signature actually does contain a date:

gpg: Signature made 11/21/03 08:53:02 using DSA key ID CD4C0D9D

-fren

SCO Again!...

Obviously SCO are trying to break in and steal the source to prove once and for all that Linux has stolen their patents!

;)

dave

Tech stuff [homelinux.net]

Re:SCO Again!...

No no. They are trying to break in to *insert* patented code into Linux code, so they'd have a leg to stand on in the court ;)

That explains

Why my apt-get was failing from people.debian.org last nite. Not to mention why debian.org was down. :(

Re:That explains

Thanks for that insightful interpretation of events, Captain Obvious.

You should be using...

Why my apt-get was failing from people.debian.org last nite. Not to mention why debian.org was down. :(

Funny, my apt-get using h4x0r3d.debian.org was working perfectly....

apt

Of course this raises the whole issue of apt-get. We all rely on apt-get update && apt-get upgrade, all it takes is someone to compromise the servers and insert a backdoor

Re:apt

After RedHat dropped their free line (I was just paying for RHN access) I have been contemplating going to Debian for my servers and suse for desktops or some other scenario. Debian packages and apt-get were primary reasons for considering that distro as my next platform. I dont want to say I am scared off by this but it does remind me that I have to put more thought into how to deal with these things. I had simply trusted RHN and the PGP signing of their RPMs, which may have been a little foolish.

I do have to say that I am still happier with Debian broadcasting this incident as loudly as possible rather than the corporate tactic of hushing it up (I know of a few companys that have done just that). Thanks for the open honesty Debian!

Signatures?

Are deb's signed? (I'm not that familiar with debian but I'd imagine they are) If so then just tell apt-get to not install debs that don't match a known signature...

Re:Signatures?

yep, GPG signed... the public keys of all the developers are avalible on http://keyring.debian.org normaly, and it still appears to be up anyway. There is also a debian package which contains all the keys too

Re:apt

apt-secure [debian.net] uses strong cryptographic methods to verify the authenticity of packages in the archive. It may be the default apt-get for sarge, depending on man-power issues.

Re:SO MUCH FOR YOUR SECURE OPERATING SYSTEM

It's not a hole, though. So far we only know it as a login/password that was comprimised. Any system no matter how secure is susceptible to that. Most of Microsoft's holes are much different - they're exploitable and are available from the default recommended installation, meaning the computer grandma bought for Bobby is susceptible and will probably never be patched.

Digital Signing of Packages?

This is the second time this has happened to a big open-source project (the first being the GNU servers a while ago). All packages by both groups are "md5" signed, which is supposed to protect against malicous hacking. However if the root server is comprimised, this doesn't help. Companies (including at least Microsoft, and the people who make ad-aware) who distribute files over the internet sign them with an RSA (or similar) key, and the computer which does this signing is kept disconnected from the internet. For such large projects which are installed by millions of people, might a similar system not be a good idea?

Re:Digital Signing of Packages?

MD5 sums are used for the contents of packages, but packages may only be uploaded and processed by the build system if they're correctly signed.

So yes it's not trivial to backdoor a package - unless you're already a Debian Developer...

Re:Digital Signing of Packages?

Don't be certain that digital signing is such a cure.

The person operating the non-networked signing machine still needs to be sure that what-it-is-that-they-are-signing is what-it-is-supposed-to-be.

Now how does digitial signing on a non-connected machine help you know the source wasn't tampered with?

Nobody's asking you to trust the keyserver

Then the next point of failure becomes the keyservers. How do you know you imported a good key, and that the keyserver hadn't been compromised when you did it?

PGP keyservers (unlike, say, Kerberos KDCs) are completely untrusted. Anyone can upload any key to a keyserver. And downloading a key from a keyserver implies nothing about that key.

To verify that you have a valid key, you have to rely on the web of trust. Basically, if a key is signed by someone whose key is signed by someone [recurse through however many levels you are comfortable with] whose key you have personally inspected, then the key can be assigned a trust metric based on how reliable you consider that chain of signatures to be. (Basically, how much you trust the integrity and acuity of the people controlling the chain of signatures.)

PGP and GnuPG have supported this infrastructure from Day 1. Asking people to trust an arbitrary third-party public keyserver was never in the plans.

How long will it take?

How long will it take for the few MS fanboys around to say that this why Windows is better? Let me pull a Rumsfield (pre-emptive retaliation, that is...). Everyone gets comprimised once in a while. At least Debian is open about it, and not sitting on an insecure system because it's more profitable to let a bad product go then to risk bad press from releasing a security bulletin.

Re:How long will it take?

Password stealing is pretty OS independent.

So this compromise, whilst undenyably bad, isn't really going to show much about Debian, or Windows.

Would Microsoft announce that it was compromised?

I doubt that Microsoft (or any commercial software company) would publically annouce that it had been compromised. The source code processes at Microsoft are opaque -- nobody knows exactly who is putting what into the source code. If hackers, goverment officials, RIAA, etc. are modifying Window's source, nobody would be the wiser. In contrast, the openness of open source development creates an audit trail of who did what to the code (assuming the version tracking and submission system is not compromised).

Transparency is a prerequisite for trust.

Re:Would Microsoft announce that it was compromise

In the days before the Pure Food and Drug Act, it was considered "nobody's business" what was in the food we eat, either; you just opened the can and accepted whatever was in there. Times change.

Re:Honestly...

I hate to say it, but Microsoft's haven't been compromised, and they're the bigger target.

Not true. [computerworld.com]

Everyone here knows if windowsupdate.microsoft.com had been compromised, people would be droning on about how it's some sort of illustration of Microsoft's security.

Their update server wasn't compromised, but the debian archive also wasn't compromised in this case. But, yes, we have to work harder to make our servers secure. And we will never reach the point were our systems will be unvulnerable. So what is your point? You complain that there aren't enough anti-oss-trolls here?

Re:Has a Microsoft release ever been compromised?

"a Microsoft release has never been delayed because one of their servers were compromised."

I don't know if this delayed a release, but -- in October 2000, the news broke that Microsoft's internal network had been cracked for three months.

(Debian made this announcement in 24 hours.)

Read for yourself:

Microsoft Cracked [slashdot.org]

...the Wall Street Journal article which apparently broke the news - it's the most complete. What's known - the passwords were being sent to St. Petersburg, Russia. They probably had access for about three months.

"LONDON (CNNfn) - Hackers gained access to some of Microsoft Corp.'s essential product secrets, the world's most powerful technology company said Friday, acknowledging a security breach that is a major embarrassment for the software company..."

"The Wall Street Journal said security employees had discovered that passwords used to transfer the source code behind Microsoft's software were being sent from the company's computer network in Redmond, Washington, to an e-mail account in St. Petersburg, Russia. Microsoft said it was making sure hackers could not use the stolen source code to change commercial software used by businesses, governments and consumers."

Hearing the news,

...thousands of slashdotters flocked to Netcraft website to check whether debian.org was running on IIS.

Makes you wonder

It really is impressive for me how honest some organizations have been about admitting system compromises (Debian, ProFTP, GNU.org).

As someone who works with networking security, I know lots of business servers get compromised regularly. Everyone hides it because it's embarassing for a business.

This makes you wonder how often other 'critical systems' get compromised, and get fixed without any public reports. Government computer systems get regularly compromised after all. But I'm sure so do vital Microsoft, IBM, systems, etc. Windows Update, anyone?

Re:How in the world...

Yes Debian's machines run Debian, this breakin wasn't anything to do with the software installed upon the box, as it was due to a password compromise.

If anything it's more embaressing that somebody lost their password than that the software wasn't up to date.

Re:How in the world...

Of course, we shouldn't jump to conclusions until we get more information, but really, I don't see an easy way out of this.

Why should you? They were cracked. The bad thing has already happen, so there is no easy way out. However, there *is* a *right* way out. And that includes telling people what they know as quickly and effectively as they can. Too much information too early can be a bad thing.

In short: have a little faith that they're dealing with this correctly, unless you've run a massively-used public box for years without a single compromise.

-Rob

Re:How in the world...

> I noticed that nowhere did they mention just *how* they were compromised.

They will when it's known. They fe