I suppose at some point this will become an arms race - that is, telemarketers will be openly hostile and completely annoy every single person they call. I'll enjoy it!
Security breach on kernel.org
Earlier this month, a number of servers in the kernel.org infrastructure were compromised. We discovered this August 28th. While we currently believe that the source code repositories were unaffected, we are in the process of verifying this and taking steps to enhance security across the kernel.org infrastructure.
What happened?
- Intruders gained root access on the server Hera. We believe they may have gained this access via a compromised user credential; how they managed to exploit that to root access is currently unknown and is being investigated.
- Files belonging to ssh (openssh, openssh-server and openssh-clients) were modified and running live.
- A trojan startup file was added to the system start up scripts
- User interactions were logged, as well as some exploit code. We have retained this for now.
- Trojan initially discovered due to the Xnest
/dev/mem error message w/o Xnest installed; have been seen on other systems. It is unclear if systems that exhibit this message are susceptible, compromised or not. If developers see this, and you don't have Xnest installed, please investigate. - It *appears* that 3.1-rc2 might have blocked the exploit injector, we don't know if this is intentional or a side affect of another bugfix or change.
What Has Been Done so far:
- We have currently taken boxes off line to do a backup and are in the process of doing complete reinstalls.
- We have notified authorities in the United States and in Europe to assist with the investigation
- We will be doing a full reinstall on all boxes on kernel.org
- We are in the process of doing an analysis on the code within git, and the tarballs to confirm that nothing has been modified
The Linux community and kernel.org take the security of the kernel.org domain extremely seriously, and are pursuing all avenues to investigate this attack and prevent future ones.
However, it's also useful to note that the potential damage of cracking kernel.org is far less than typical software repositories. That's because kernel development takes place using the git distributed revision control system, designed by Linus Torvalds. For each of the nearly 40,000 files in the Linux kernel, a cryptographically secure SHA-1 hash is calculated to uniquely define the exact contents of that file. Git is designed so that the name of each version of the kernel depends upon the complete development history leading up to that version. Once it is published, it is not possible to change the old versions without it being noticed.
Those files and the corresponding hashes exist not just on the kernel.org machine and its mirrors, but on the hard drives of each several thousand kernel developers, distribution maintainers, and other users of kernel.org. Any tampering with any file in the kernel.org repository would immediately be noticed by each developer as they updated their personal repository, which most do daily.
We are currently working with the 448 users of kernel.org to change their credentials and change their SSH keys.
We are also currently auditing all security policies to make kernel.org more secure, but are confident that our systems, specifically git, have excellent design to prevent real damage from these types of attacks.
---------- Forwarded message ----------
From: J.H. <warthog9@kernel.org>
Date: 2011/8/29
Subject: [kernel.org users] [KORG] Master back-end break-in
To: users@kernel.org
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Afternoon Everyone,
As you can guess from the subject line, I've not had what many would
consider a "good" day. Earlier today discovered a trojan existing on
HPA's personal colo machine, as well as hera. Upon some investigation
there are a couple of kernel.org boxes, specifically hera and odin1,
with potential pre-cursors on demeter2, zeus1 and zeus2, that have been
hit by this.
As it stands right now, HPA is working on cleaning his box, and
I'm working on hera (odin1 and zeus1 are out of rotation still for other
reasons), mainly so that if one of us finds something of interest, we
can deal with it and compare notes on the other box.
Points of interest:
- - Break-in seems to have initially occurred no later than August 12th
- - Files belonging to ssh (openssh, openssh-server and openssh-clients)
were modified and running live. These have been uninstalled and
removed, all processes were killed and known good copies were
reinstalled. That said all users may wish to consider taking this
opportunity to change their passwords and update ssh keys (particularly
if you had an ssh private key on hera). This seems to have occurred on
or around August 19th.
- - A trojan startup file was added to rc3.d
- - User interactions were logged, as well as some exploit code. We have
retained this for now.
- - Trojan initially discovered due to the Xnest/dev/mem error message
w/o Xnest installed; have been seen on other systems. It is unclear if
systems that exhibit this message are susceptible, compromised or not.
If you see this, and you don't have Xnest installed, please investigate.
- - It *appears* that 3.1-rc2 might have blocked the exploit injector, we
don't know if this is intentional or a side affect of another bugfix or
change.
- - System is being verified from backups, signatures, etc. As of right
now things look correct, however we may take the system down soon to do
a full reinstall and for more invasive checking.
- - As a precaution a number of packages have been removed from the
system, if something was removed that you were using please let us know
so we can put it back.
- - At this time we do not know the vector that was used to get into the
systems, but the attackers had gained root access level privileges.
That's what we know right now, some of the recent instabilities may have
been caused by these intrusions, and we are looking into everything.
If you are on the box, keep an eye out, and if you see something please
let us know immediately.
Beyond that, verify your git trees and make sure things are correct.
- - John 'Warthog9' Hawley Chief Kernel.org Administrator -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAk5a5U0ACgkQ/E3kyWU9dif+1ACfYPlgq/keFrFO77AmQVduKGwx TAcAnRAu6nHt74+5aC+fPeb8aT0hcy2K =Semd -----END PGP SIGNATURE----- (I can't pretty-print this any better due to slashdot's lame lameness filter.)
Friction is a drag.
