CAIDA Released Code-Red Worm Post Mortem

davidu writes "David Moore at CAIDA (The Cooperative Association for Internet Data Analysis) was monitoring an entire /8 network while the code-red worm traversed the net. His findings are really interesting and show just how swiftly code-red moved across the net and infected hosts. It was the sheer stupidity of the worm's creator and the skill of some network admins which limited the worms attack and DoS potential. note: Check the graphs, these pictures really do tell a thousand words."

Re:Don't be a jackass

There is about 3 security bulletins from M$ per week.

Exaggeration. While this was true in the past, the rate of such bulletins has been slowing. I've received three for the entire month of July so far.

This patch in question requires SP1 to be installed as well. If the IIS server was up withoug SP1 then that requires 2 reboots to get the server patched.

And, as others have said, any system administrator worth his salt has already installed SP1 for Windows 2000. Therefore, it's really only one restart.

In many cases, the admins are overworked and cannot get to every patch all the time.

Indeed. That's why you put in extra hours to fix things. MS may not be the best server software in the world, but any competent MS system administrator applies the patches as they come out, maintains a reasonable schedule, and tells the bosses flat-out, "I'm installing this patch at such-and-such a time, and that's all there is to it." Few employers are willing to fire a system administrator who's doing their job.

Personally, I had 1 of 3 IIS servers at my job vulnerable and it was 'ploited.

Yes. Imagine how you would have felt if you'd stayed on top of it. It's easy to say that you don't have time to install the patch, but on any reasonable server-level machine, the patch takes maybe five minutes to install, and most of that is spindown/startup time.

I have enough on my plate then to jump at every damn MS Security Bulletin.

If this is your attitude, you need to find another line of work. I wouldn't want you administering anything of mine.

If you actually care about what you do, then you MAKE the time. Explain to people what you're doing. Encourage them to understand what's involved. Tell people to piss off, you're saving the company.

There are just so damn many of them!

39 this year. That averages to slightly more than one per week thus far. This is a lot, to be sure, but it is not "too many." The thought "too many" should be followed by the thought, "What are my alternatives?"

If you're that peevish about MS product security, then don't use MS products.

I am overworked as it is yet my CEO still asks "What exactly does he do again?"

Then quit. Get a job elsewhere. Do something else.

Re:I see a nice research paper in this

I don't know what bothers me worse: The fact that the authors of the paper listed here think they've discovered some great fundamental truth, or some slashdot readers think in such narrow tracks. It's no great secret that IBM Research [ibm.com] exists, or that they spend a great deal of money, time, and expertise working on issues that we run into all the time [ibm.com]. 5 minutes with Google could have easily uncovered it.

I posted to BugTraq that the published curves for Code Red infection rates looked very much like traditional biological infection rates, and was soundly rebuked in emails by people who obviously knew better, except they didn't.

"Those who forget the past are condemned to repeat it." -- Santayanna


...phil

I see a nice research paper in this

Beyond what the authors have done, this research could be used as a basis to compare the spread of virii in fixed pool, whether biologically based or network based. While there's been a lot of speculation on the spread of computer virii before, this appears to be the first study with hard numbers that could be used for comparison.

Sure, the results aren't that surprising, but it's still an interesting comparison.

Do they want to eat their cake, too?

for many Fortune 1000 organizations, patching is a bad thing. They want stable systems and have a rigorous change control process to guard against problems.

Great. Do they have an artificially intelligent firewall, too? That's what it's going to take to allow people to run software with known security holes for very long.

Does anyone else remember the worms that were attacking unpatched Red Hat systems ~3 years ago? It was six months between the time the exploits were discovered/patched and the time that the worms started making their rounds. A more recent Red Hat attacking worm came out something like 3 months after the security holes it exploited were discovered. Now we've got an IIS security hole, with a worm exploiting it within a month.

Do you not see where this is going? We're at the point where virus/worm authors aren't just reusing each other's code, they're talking about writing modular hostile code in the first place! Take a "worm kernel", load in modules to install back door A, autonotification service B, and brand new exploit C, and send it off to the internet the same damn day you discover a new buffer overflow.

This is coming soon, and if you have computers hanging out on the internet, you need to be ready for it. Don't give me any BS about "rigorous change control". If you want to think of it in those terms, think about this: Running known exploitable, publically accessable software will cause your computer systems to undergo uncontrolled changes without your approval!

Throwing many MS OS/App patches into the mix without testing the effects of the patch on your systems environment is just as foolish as not installing the patch.

No, it really isn't. What's the worst that buggy MS patches can do to you, reformat your hard drive? Not installing the patch can result in your data being published to hostile destinations, your passwords being sniffed, other systems on your network being attacked by the compromised unpatched system, your network being flooded by the compromised system, and your business being brought to a halt for days while you explain to the feds why your computer was being used to try to crack *.fbi.gov. Oh, and for kicks, the attacker/worm might reformat your hard drive afterward anyway, to cover his tracks.

Re:You can't blame them entirely

Simple. If a customer's machine is responsible for further spreading a virus, worm, etc. the ISP should CANCEL the customer's account without a refund. People would be more responsible if irresponsibility affected their wallets.

Posturing

It was the sheer stupidity of the worm's creator and the skill of some network admins which limited the worms attack and DoS potential.

Riiiggght. This is the second time Code Red has been mentioned on Slashdot with a reference to the "stupid" author. Compared to the skilled network admins? What, the ones who let Windows NT boxen on their network? The ones who got HACKED by the silly virus author? Yeah, they're skilled all right, truely elite.

Mock the author as you will, but the fatal errors in Code Red were choices that the author made. His options for those choices could have been stopped, too. It wasn't really the stupidity of the virus author that saved whitehouse.gov, but the vigilance of some people doing things that might be illegal under the DMCA or some other law in the near future.

Remember that the next time you're feeling elite, yourself.

Re:IIS can be restricted and protected

It's a bit interesting (I tried that mod, too). Apple has detailed specs of what a legitimate Mac application should look like. And the Apple applications were notorious for ignoring the rules.

MS has rules for how a Windows application should act, and the MS applications are even worse than most DOS application about following those rules.

I wonder if Linux will follow this tradition ... O, wait, there is not Linux company. I guess that the LDP will be honored :-).

(LDP : Linux Directory Plan? Or do I have the wrong acronym?)
Caution: Now approaching the (technological) singularity.

What the hell are they waiting for?

Where are the truly destructive worms/viruses/trojans/etc.? I'm really surprised no one has written anything that would forward itself along, then wipe out the HD (no random chance BS), or something like that ..

And these guys seriously need to hook up with someone who knows English .. the grammar errors in the e-mail are usually enough to tip me off it's a virus to begin with, that's what I guessed about SirCam before I really knew what it was ..

Absolutely correct

From the research:

Again, 359,104 hosts were compromised in approximately 13 hours. Although the growth was slowing, had the worm not been programmed to stop spreading at midnight, additional hosts would have been compromised. The infection rate would have continued to decrease once the vast majority of vulnerable machines were infected. We speculate that the memory resident status of this worm would have allowed reinfection of many hosts.

All it takes is another version which doesn't limit itself, and the problem explodes. As it is, there was a nice easy way to stop the worm (once it stopped itself). If the worm had not stopped itself, I'm skeptical that it would have been nearly as easy to deal with the infection.

---

Prelude

It makes you wonder where all the truly devious virus writers are.

If, in the case of SirCam, files were posted to an unmoderated news group instead of e-mailed randomly then the authors could retrieve them anonymously.

Add in the ability to distinguish victims (such as hosts only on a certain domain); to quietly terminate itself if the victim isn't on "the list"; and stick to a specific task instead of just spamming and destroying -- you will have something truely devistating.

It makes me wonder what we AREN'T finding and what ISN'T getting the headlines.
--
Charles E. Hill

Version 2.0

It was the sheer stupidity of the worm's creator and the skill of some network admins which limited the worms attack and DoS potential.

I'm sure that version 2.0 of the worm will fix all of the problems.

Bob Cringely

Wrote about the coming DDoS from Hell [pbs.org].

Wait til August 1st

I'm surprised that no one is mentioning that the random infection part of Code Red is programmed to restart on the 1st of *every month*. Sure, by changing the IP of whitehouse.gov and short circuiting packets destined for the old IP to the bit bucket, the attack phase will never be a problem.

However, since it appears the number of infections capped at about 359,000 machines, I would venture that at least a quarter of those machines will not be repaired/rebooted by August 1st. If the number of infections went from zero to 359,000 in a couple of days at most, imagine what kind of storm is going to kick off on August 1st when nearly 100,000 machines restart the infection phase of the worm! How long will it take for the estimated 6 *million* vulnerable IIS servers to be patched?

Just for the sake of gloom-and-doom, how long will it take before the Internet only becomes usable between the 20th and the end of each month, due to Code Red infection storms between the 1st and the 19th? I don't think the core Internet routers can perform stateful-enough inspection as to route "Code Red infection" attacks to /dev/null. Perhaps that would drive enough white hat hackers to spread a repair worm, and start that whole argument all over again.
--
Steve Jackson

Re:Unpatched version of server software

The insurance adjuster idea is a good one, but I don't agree with the patch policy limitation. Instead, give the policy a rate structure that makes it *very* appealing for an organization to have a dedicated security person/department on hand (and not just a part time guy in IT).

As for the law and patching, you need to realize that for many Fortune 1000 organizations, patching is a bad thing. They want stable systems and have a rigorous change control process to guard against problems. Throwing many MS OS/App patches into the mix without testing the effects of the patch on your systems environment is just as foolish as not installing the patch. For some, applying a patch to server software is a several day process!

Re:The world is safe again ...

2. Explained his/her dastardly plan in detail to the heros before killing them

Hi! How are you!

I send you this file in order to have your advice.

[Attachment: Dastardly Plan Details.doc.pif]

k., who's gotten about a dozen of these so far.
--
"In spite of everything, I still believe that people
are really good at heart." - Anne Frank

Disturbing thought

Take a look at the domains that were the most-infected -- they were, by and large, cable modem providers, and the study concludes that home and small business users (read: Microsoft's target market for most of their products) were responsible for most of the worm's spread.

It's really disturbing to think that the Internet's stability rests on the shoulders of these people, half of whom probably don't even understand the concept of keeping up-to-date with security patches.

The ironic thing is that this tide is probably being held back by the fact that in order to "legitimately" run a server off a broadband connection, you generally have to pay through the nose, meaning that those who don't have a vested interest or Daddy's money need not apply.

Disturbing all around, really...

Re:IIS can be restricted and protected

At the risk of being slightly off topic...

Changing anything that Microsoft considers 'default' or 'normal' can be a problem, even when the change is relatively easy to make. In your example, I have a feeling that if you installed any additional software to work with IIS, especially MS software, it would have issues with your simple change. It just assumes that everything is the default, even if it could just check the registry during install.

To make myself a little bit clearer (while my coffee is still kicking in this morning), I'll give an example. I am a command line user, even in Linux and Windows. Try using Program Files in a command line path. It gets very, very repetitive. So I changed it to Programs. Registry search and replace, rename, a couple of other things. Yes, there is a registry key for the location of Program Files, and properly written software looks for it during an install or run. But try to install a a patch, or an upgrade, or anything else, and watch your Program Files directory magically reappear. The assumption is that nobody changes it, so Program Files is hard coded.

My point? Even when MS leaves a way to change things, they often don't honor it. So the harder you try to customize or secure a system, the more you have to work to make sure that you haven't broken something else. A sad state of affairs, it is.

Re:The world is safe again ...

Brought the heros to his/her secret mountain lair to kill them personally rather than letting a henchman do it at great distance

This is a damned-if-you-do-damned-if-you-don't situation. If you order your henchmen to do it, they will certainly screw it up, and, depending on the movie rating, will be severely injured to killed.

At least if you have your henchment bring the hero(es) to the secret lair, you don't have to pay out as much disability or have as high employee life insurance. This is why usually contractors are brought in, not because they really are the badest killers from the four corners of the earth, but because by going corp-to-corp, you won't impact your premiums when they are killed. Plus it keeps employee morale up.

Re:DoS Attacks

Simple, just have Jon Katz write all the articles posted.

Don't be a jackass

Speaking of being a jackass... don't blame it completely on the admins either. There is about 3 security bulletins from M$ per week. This patch in question requires SP1 to be installed as well. If the IIS server was up withoug SP1 then that requires 2 reboots to get the server patched. In many cases, the admins are overworked and cannot get to every patch all the time. Sure, the admins should be able to get the patch on before hell breaks loose but hindsight is always 20/20.

Personally, I had 1 of 3 IIS servers at my job vulnerable and it was 'ploited. Of course, when it the Code Red worm infected that server, the server took out one of my 2500 series Cisco routers. That was fun since it was still too early in the day to know that it was indeed the worm causing the problems. I am the only IT person here, supporting 75 users, 17 servers, 100+ workstations. I do support, net admin, and IT department management. I am currently upgrading the corporate website, doing a software audit, a hardware audit, reconfiging our routers, I have 30+ helpdesk issues in my queue and I am late on 4 projects. I also advise our development team on network related aspects and I am trying to put up a new FTP server, backup server and mail server. I have enough on my plate then to jump at every damn MS Security Bulletin. There are just so damn many of them! I am overworked as it is yet my CEO still asks "What exactly does he do again?".

In the future will I put a little more time at getting the patches on the IIS servers when they come out? Sure will. Did I learn a lesson? Yes. Did my company learn a lesson? Nope. Not until I leave this place and they have nobody around...

You can't blame them entirely

It's really disturbing to think that the Internet's stability rests on the shoulders of these people, half of whom probably don't even understand the concept of keeping up-to-date with security patches.

I think it's safe to say that most people on Slashdot are not only competent enough to apply patches, but interested enough in computers (for work or a hobby or whatever) to actually do it.

But we're not a typical cross-section of the public. People are used to buying something and having it work. They don't need to patch their TV every couple of months to prevent people abusing it, and they just don't (and probably never will) see why they should do this for their PC, which is just another appliance (to them at least). And I'll bet that 95% or more of Slashdotters wouldn't fix their car themselves if it started burning a lot of oil - it's all a matter of whether you're willing and able to do the job.

The only way you're going to stop people like this propagating worms or virii or whatever in this manner is by taking that need for vigilance out of their hands. Quite how you do that without infringing on their privacy is beyond me. But just think about the fuss that would be kicked up here on Slashdot if Microsoft wrote it's software to require MS full access to it's OS at all times over the phone line under the pretext of helping home users keep their machines up to date.

Don't criticise the regular consumers unless you've got a better solution. And I don't count banning them from the net as better (even if it does have a certain appeal).

Re:Openness Good...

who's to say that hubris won't set in?

That's very true actually. I mean, I'm pro *nix, anti Microsoft/Windows but lets not forget that buffer overflows come from the use of the crappily designed stdlibc which is only still a standard because of years of acceptance in the Unix community.

I mean, sure it's the developers fault for using these functions but as a community, we should have kicked scanf and friends out decades ago. Compilers should complain if you use them. Heck, they should refuse to use them unless you define #NOTTOBEUSEDONAPRODUCTIONSYSTEM or something.

Rich

Re:Don't be a jackass

Sorry, chief. 1: This particular patch didn't require a reboot. 2: this particular patch wasn't required UNLESS YOU'RE USING INDEX SERVER. The five second work around was to remove the script mapping from IIS that would pass the request to the index server stuff, which is where the problem actually lived.

Goodbye NT

How long will it take for the estimated 6 *million* vulnerable IIS servers to be patched?

More to the point, how long will it take for people to switch to a web server platform that isn't fundamentally insecure? I'm not a "Free" software true-believer, or a compulsive Microsoft basher. But the object fact is that NT two of the biggest symptoms of an insecure system:

• A bloated API. The implies complexity, which guarantees a continuing stream of new exploits.
• No outside review of its design. You can't take a software vendor's word for it when they claim their product is secure. Whatever the economic shortcomings or moral strengths of open source, it does seem to be the only way of guaranteeing the absence of undiscovered exploits.

As I've said before, the day will come when it will be illegal to operate an insecure system on the public internet. Perhaps sooner than later.

Side note: what's with wasting all that bandwidth on Quicktime animations? The Flic files are a fraction of the size, and run on the same viewers.

__

Available animation formats

The animation is available in three formats: flipbook/flic (207k), QuickTime (13.4 MB), or as an animated gif (4.1 MB) [...] Note: The recommended way to view the flipbook format is to use xanim on a Unix platform, or QuickTime Player 5 on Macintosh and Windows boxes. Use the "open URL" feature of a QuickTime player and paste in the URL.

how much you want to make a bet that a lot of folks are going to grab the 13 meg quicktime file?

The .fli file works just fine.

Check out the Vinny the Vampire [eplugz.com] comic strip

IIS can be restricted and protected

It is Possible to run a secure NT Based Web/SQL server. The problem is that MS makes everything run as the system acocunt on the machine by default. Most people don't change the defaults. Things get further complicated if you do change the account that these services run as, due to the fact that nowhere do they tell you all the things that these services need access to, or to talk too. Then things get even worse as they make assumptions in upgrades, patches, and addons that you are running as system.(This is a major problem when it comes to frontpage at times) The fact still remains though that you can infact change the service account that these services run as, to a different account and then restrict the access these accounts have to other parts of the system. For instance, only the IIS, service account has access to the SQL server that backends it. Then you make the service account for IIS only a local account on the Web box, with no global domain access. Then you take the actual logon rights away from that account, then restrict it from access to anything outside of wwwroot of the IIS box. Then give ownership of all the files that make the web site to a different account and give the IIS account read only access. The same can be done with the SQL account, only a local account, no access to anything on the box. it can be done it just takes work, and Most M$ admins are to lazy to do it.

Microsoft Bundles Worm with IIS

BBspot [bbspot.com] has a great satire of a new bundled feature for IIS from Microsoft.

Take a look at Microsoft Bundles Worm with IIS [bbspot.com]!

Give a man a fish and he will eat for a day.

Re:Priorities

Personally, I had 1 of 3 IIS servers at my job vulnerable and it was 'ploited. Of course, when it the Code Red worm infected that server, the server took out one of my 2500 series Cisco routers. That was fun since it was still too early in the day to know that it was indeed the worm causing the problems. I am the only IT person here, supporting 75 users, 17 servers, 100+ workstations. I do support, net admin, and IT department management. I am currently upgrading the corporate website, doing a software audit, a hardware audit, reconfiging our routers, I have 30+ helpdesk issues in my queue and I am late on 4 projects. I also advise our development team on network related aspects and I am trying to put up a new FTP server, backup server and mail server. I have enough on my plate then to jump at every damn MS Security Bulletin. There are just so damn many of them! I am overworked as it is yet my CEO still asks "What exactly does he do again?".

And you find time to read Slashdot? Well, at least you have your priorities straight.

Unpatched version of server software

Around 10:00 UTC in the morning of July 19th, 2001 a random seed variant of the Code-Red worm (CRv2) began to infect hosts running unpatched versions of Microsoft's IIS webserver.

If I were an insurance adjuster trying to insure peoples' information technology assets, I would have my own experts supervising everyone who was on the insurance plan to ensure that they patched their fucking software.

Or I would make it against the law not to patch one's software, similar to the laws ensuring the vaccination of children, and for the same reasons; such an epidemic, viral or virtual, delivers a powerful blow to our economy and is a matter of national security.

Eerie parallel with biological epidemics

If ever there was a more graphic proof why monopolies are bad...

What I find interesting is the parallels with biodiversity. One of the argument for biodiversity, especially in agriculture, is that a wide variety of species will slow the growth of any disease or epidemic. If everyone planted the exact same species and variety of wheat, a single organism could wipe out the global harvest; but if everyone used whatever species or variety they felt like, an opportunistic organism's growth would be blunted. The organism can't adapt and infect to a hundred varieties of a crop, so it will try to infect unideal hosts and fail.

This same argument can be said for software. If everyone uses the exact same software from the same company, then an opportunistic hacker or virus could rapidly take over everything; but if there were more companies and products out there, then the virus/worm would either have to learn how to hack a dozen or more different systems, or it is limited to growth among one particular system. So if MS gets its way, we'll get computer equivalents to AIDS and Ebola creating pandemics of worms and viruses. But if there were more competitors, then no single worm or virus could ever pose much of a threat.

The world is safe again ...

It was the sheer stupidity of the worm's creator and the skill of some network admins which limited the worms attack and DoS potential.

Once again, evil is thwarted because, just as on television, the villans are incompetent while the virtuous are strong and intelligent.

I wonder if the virus author also committed any of the following classic villan errors:

1. Brought the heros to his/her secret mountain lair to kill them personally rather than letting a henchman do it at great distance
2. Explained his/her dastardly plan in detail to the heros before killing them
3. Arranged for a dramatic but overly-complicated and easily escapable death for the heros
4. Once the heros escape, get a squad of elite ninjas to track them down, but have the ninjas attack one at a time so as to ensure defeat in spite of superior numbers

So, the world is safe again ... but ... for how long?