Slammer Worm Slams Microsofts Own

MondoMor writes "Microsoft's forgot to patch some of its own servers to protect it from the months-old vulnerability exploited by the Slammer Worm, reports C|Net. Oops. Apparently Redmond's network was hit pretty hard. Just goes to show that no matter who you are, you'd better keep your apps patched." Update: 01/29 01:59 GMT by T : And if you're running systems which might be affected, take note: whitehorse writes "The Microsoft KB article for the Slammer patch found here has an incorrect URL for 'Download the patch' referring to KB Q316333 which is only a handle leak fix. The real patch may be found later in the article."

Microsoft didn't patch all their INTERNAL servers

[wbm6k]

The article I read (on yahoo [yahoo.com]) states the unpatched servers were all on the internal network, not the internet, and that they were in use by researchers within microsoft.

Let's not jump too quickly on the bash microsoft bandwagon for that. (Of course, if they just did enough testing and didn't release buggy, vulnerable software in the first place...)

Re:Zoiks!

[Anonymous Coward]

automatic update doesn't work with SQL server. you have to do patches the "old" way (unzipping files, renaming files, prayer), which is probably why so many novice admins never applied the patches.

Re:SQL Server

[the_Bionic_lemming]

Actually, the vulnerability exists for sql 7 as well. If you haven't patched it's only a matter of time.

Re:I wonder how long...

[jamesdood]

The thing to remember is this worm infected any machine running the MSDE (A scaled down MS-SQL server) So if you were running Access or Office 2000 or MS Visual Studio 6, or even Visio 2000 you could be affected by this. Most end users don't even know that they would be vulnerable and the statement "This particular worm largely ignored home and personal computers, due to the product it infects" is false. It also seems to have had an effect on certain Cisco routers. Not fun but you can't just blame "Poor Admins" as the culprits for the virualance of the worm.

Re:Zoiks!

[questionlp]

Please mod this parent up.

There isn't only no way to get SQL Server patches from Windows Update, but (as the parent mentioned), the steps required to update SQL Server and the Desktop Engine (MSDE) is a royal bitch and some.

For example, to apply any hotfixes or cumulative patches for SQL Server 2000, you must download the package, extract it, backup the SQL Server install directory and databases, manually copy over DLL files and other updated binaries, execute the SQL query files included in the patch (one at a time, in a certain order... MSDE users need to use the command line interface for it since there is no GUI provided), then pray that everything is okay and start SQL back up.

Worm's damage surprises experts -- takes out ATMs

[aengblom]

My rejected submission -- more details, but a bit long. The big news in my mind was not the microsoft bit--it was that ATM machines were unavilable because of the worm.

~~~

The worm that slowed the internet to a crawl over the weekend apparently did more damage than most originally believed. On Monday, many companies were still struggling to clean up. Financial companies and airlines seemed to be hit most acutely. Many web sites that manage payments and check loans were inaccessible. Inexplicably--and really inexcusably--some ATMS were also unavailable. Investigators are also struggling to pinpoint the worms starting point, but are having little success because it took off so fast.

Apparently similar code was released by David Litchfield of NGS Software Inc a few months ago. Virus "author," "Lion" credited Litchfield's code.

The Washington Post has an AP [washingtonpost.com] story up as well as this [washingtonpost.com], which is older but has some additional details. The kicker to all this--the worm hit one year after Microsoft launched its "Trustworthy Computing." That and even some of Microsoft's own computers were hit [nytimes.com] (NYT Reg. Req.).

(Yep, still bitter ;-) )

Re:Despite what the apologists say

[thebatlab]

> Want to find a list of needed patches for a Microsoft product? Hope you have a few days for searching the endless volumes of technet or msdn-- hope you find everything.

I usually just go here rather than searching endless volumes....

http://www.microsoft.com/technet/treeview/default. asp?url=/technet/security/current.asp [microsoft.com]

Re:Speaks volumes for their policies...

[Wansu]

It is in fact blaming the victim for the software's flaws.

Yep. The same can be said for clicking on virus laden emails. Back when the "I love you" email virus was making the rounds, some MSCE type sent out an email scolding people for clicking on bad emails at the comapny where my wife worked. The next day, her inbox had 50 some emails from him where he'd clicked on a bad email.

Later the same week, our IT dept deployed the last anti-virus patch. I set it off looking at comments on Slashdot where somebody had posted the Word Basic macro that was doing all the dirty work. The dern virus scanner was keying off the macro source. That caused a bru-ha-ha.

Re:I think you're running the update seriously wro

[questionlp]

I'm not talking about Service Packs... but hotfixes, like the one for MS02-056. Of course, they provide an additional tool to help automate the install process of hotfixes (here [microsoft.com]) that make it a bit easier. But before that was available, take a look at the previous cumulative patches for SQL Server 2000 and read the readme file for the install process. Not as easy as installing a Service Pack, no?

Re:Big Surprise?

[auferstehung]

MS was able to draw on (and some would say corrupt) the smart work of their research folks and of technologies that they acquired and "MS all over it" until it fit their sales and support model, which is one of the reasons that they could do something like go from "Internet-illiterate" to winning the browser war, practically overnight.

From about Internet Explorer: "Based on NCSA Mosaic. NCSA Mosaic(TM); was developed at the National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign." Microsofts "smart work" was assimilating open-sourced non-GPL code to become internet literate. Explains why they dislike the GPL. It puts a damper on their research and innovation.

Re:Say what?

[Hektor_Troy]

Well, SQL Slammer is only a TSR-worm and isn't stored anywhere but memory, so those who actually turn off their laptops as opposed to suspend or hibernate wouldn't be carriers.

Of course - it only takes one infected client ...

Zero defects impossible; fix the fences instead

[satch89450]

Zero defects is not an attainable goal; it's too expensive and no one wants to pay for it.

This article shows just what happens when you expect zero defects in the infrastructure of a large organization like Microsoft Corporation. It's not going to happen. And before someone says I'm Microsoft-bashing, I will say that this is true for the vast majority of corporations, universities, foundations, and governments. That would include Sun, IBM, Red Hat, even the *BSD folks and LKML participants.

There is a damn good reason we won't see zero defects: employees are not measured by it. Their survival, pay raises, and promotions are based not on the number of defects they don't have, but on their contribution to the "bottom line." If you preach zero defects as Job One, then prove it by firing the people who generate defects, without exception -- including the CEO, COO, CFO, CIO, and other top brass, when they screw up.

So now that the myth of zero defects has been exposed for what it is, what do we do about it?

1. System administrators are going to have to re-think their perimeter access controls. This may require router upgrades to add processing power to support additional filtering.

2. Sysadmins who have been running "mostly-open" filter configurations may want to consider moving to a "mostly-closed" configuration: deny everything except services that have been cleared for use. Don't allow arbitrary connections. Many unknowing MS SQL servers were protected from participating in this little exercise because the firewall upstream of the desktop system wasn't allowing connections to get through, even if the desktop system had a globally-routed Internet address.

3. Computer mail order houses and computer stores should consider carefully whether they should bundle appropriate software firewall products with the computers they sell. Software configured to require the user to say "Yes, I want to make SQL server available for public access!" before 1433 and 1434 would be open.

4. We need to ask the reporters and editors of mainstream publications to be more responsible when reporting problems like Sapphire/SQL. The facts were pretty well known, and available to those who tried hard enough to get them even at the height of the packet storm, so that reporters could make their deadlines and get the facts straight. [Names of the guilty withheld, at least for now -- they know who they are.]

5. Tier 1 and Tier 2 bandwidth providers need to consider modifications to their Acceptable Use Policies to require some basic filtering of packets in both directions. These AUP changes have been discussed before; perhaps now is the time for them to go into effect:

   • Upstream packet source addresses must be verified at the perimeter such that the packet's return address points to a host in the network, and not to a random IP address or to broadcast addresses
   • Downstream packet destination addresses must be verified at the perimeter such that the packet is directed to a single host in the network, and not to a random IP address or broadcast address (other than multicast addresses, if such are allowed in the network)
   • As one drills down the levels of networks, packet source/destination verification must be done at all levels -- no exceptions (the excuse "It costs too much" doesn't wash when you consider that suitable packet filter technologies are available in both *BSD and Linux flavors, running on hardware that costs less than your standard business power lunch for four)
   • "Small services" (TCP 0:19 and UDP 0:19) must be blocked at the perimeter, both as source and destination ports.
   • A small number of other, specific ports must be blocked at the perimeter, those ports being identified as services that are intranet in nature instead of "global" services. The specific ports to be blocked should be determined co-operatively to avoid denying essential services to customers.
   • Encourate the use of VPN for interaction between two separated locations needing the above-mentioned intranet services over the Internet.
   • Encourage the use of abuse-prevention methods such as Network Address Translation on all circuits [cable operators take note] to block access to those systems that are NOT intended to be servers.

6. Update the Best Practices RFCs to incorporate some or all of these suggestions, so that Internet operators around the world can participate in solving the problem.

(N.B.: I want to point out that many USA-based cable operators are contributing to the problem by disallowing the use of NAT and VPN technologies in their apparent [alledged] quest to limit the broadband "Internet service product" to browsing and downloading files. I believe that such an attitude contributed to the problem, not the solution. I understand well the technical and business motivations for this, but I also believe that there are (U.S.) national security implications against such a policy. THINK!)

Are any of these ideas new? NO. The only new idea is to have the Lords Of The Internet use their influence over their customers to implement them more widely.

Good fences make good neighbors. The Internet is a neighborhood.

Re:Say what?

[br0ck]

Rebooting clears the worm from memory and don't most people shut their laptop off when they carry it in to work? Actually we quite a few laptops and desktops that needed patching and the users had no idea that they had MSDE or SQL server running. Windows Update doesn't notify these users that they need patches, and these users definitely include the types that would have no idea that they need to track down obscure patches that didn't even, until this weekend, have an install routine. MSDE installs by default with Visual Studio .NET, ASP.NET web matrix tool, Office XP Developer, MSDN subscription, Accesss 2002, Visual FoxPro 7/8 and can be redistributed by vendors. Even worse, MSDE and SQL server install with a blank password [cert.org], although I think it warns on install now.

Re:I think you're running the update seriously wro

[Anonymous Coward]

SP3 just came out Jan 17th, 2003. Hotfixes are, as the other guy noted, a major pain. Manually installing the dlls and keeping your fingers crossed. I know this because I recently had to use a hotfix on our production software because SP2 introduced a bug that caused SQLServer to Crash during certain replication scenarios. I was anxiously awaiting SP3 so that I could get the hotfix off our customer's machine and I didn't even know it was out until the notice that it was the fix. I also spent half of December on the phone with SQL Server support and I asked at the time what I needed to make it secure, etc. I was told that SP2 had no security holes. MS botched this one big time. People are right about service packs breaking other code. Replication is a prime example, it doesn't work the same way in the original release as it does in SP1, nor does it work the same way in SP2...hopefully I don't find out that SP3 changed things too much.

Useful script for keeping up-to-date with RPMS.

[caluml]

cd /raid/8.0/updates

wget -nd -nH --mirror --no-parent --passive ftp://ftp.mirror.ac.uk./sites/ftp.redhat.com/pub/r edhat/linux/updates/8.0/en/os/i386/ -o log
wget -nd -nH --mirror --no-parent --passive ftp://ftp.mirror.ac.uk./sites/ftp.redhat.com/pub/r edhat/linux/updates/8.0/en/os/i686/ -a log

saved=`grep saved log | grep -v ".listing"`

check=`rpm -K /raid/8.0/updates/*.rpm | grep -v "md5 gpg OK"`

if [ "$saved" ]
then
mail [e-mail] [e-mail] <<EOMAIL
New RedHat 8.0 RPMs downloaded onto `hostname`
Please update them:

$saved

$check

If there are any kernel updates, please run lilo before rebooting

EOMAIL
fi

Run this in the night some time.
When you come in, if you've got an email, run:
cd /raid/8.0/updates
rpm --freshen -vah *.i686.rpm
rpm --freshen -vah *.i386.rpm

Hey presto. Job done. And if you use Grub, you don't have to bother about running lilo.

Re:The MS security update is confusing

[Repugnant_Shit]

RTF files can be read with WordPad, which is included with Windows.

Patch + Honeywell = SOL

[Xiver]

We actually had Slammer hit us through our client's network, which was not supposed to have any "extra" computers on it. We cannot install SP3 on that internal "isolated" network because the software that runs on top of it will break. It puts us between a rock and hard place. We have to wait for Honeywell to give us a patch to fix a Microsoft bug. Its like some bizarre bad dream.

Actually No

[ravenmoon]

Microsoft incorrectly states in bulletin MS02-061 [microsoft.com] that SQL Server 7.0 and MSDE 1.0 are also affected by the worm.

While troubleshooting an issue related to the patch w/ MS phone support, the technician told me that 7.0 is not affected and the bulletin was incorrect.

It is entirely possible he was misinformed though.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:hmmm...security + patch administration

[painehope]

most of the few hundred lines of code is for a system that fetches the patches from a site that maintains a redhat mirror ( site selection by ping time ) to a local patch server, and does the comparison and selection ( w/ a somewhat optimized algorithm ), then moves old patches out of patch tree. It only took about 60 for the updating stuff. It's what I use on workstations. For servers and clusters, yes, I administer those more directly ( select an appropriate patch, drop it in a patch directory, and it is installed by autorpm ). Perhaps I should have clarified that. But I'm generally not too worried about most linux patches, as I can be sure that the patch I install for a utility or a desktop app isn't going to include something that break mysql, apache, etc. And you can use some of these package management packages ( no pun intended ) on most Unices, for example, RPM on AIX. Most of my admin experience is with Linux, but I've done some AIX + Solaris, and I'll agree that the default methods are a bitch, but they can be made more efficient and simple. And Linux is drop-dead easy to administer on workstations, pretty easy on servers, and very easy on clusters.
As to why I don't use up2date, it's because it chews up to much bandwidth ( ea. workstation hits redhat patch server ). I looked at using Ximian's stuff, but management shot it down.
And if administering w2k boxes is so damn easy, how come very few admins do it right? Proportionally quite a good deal less than their Unix counterparts? I can't verify the comments you made about w2k package management, as I quit doing windows around NT4.0/98 days, but I'll take your word on it. BTW, can this be remotely automated? But, still, how come everyone whines about how hard it is? And how come a frequent complaint ( and one I've seen w/ my site's sExchange mail servers ) is that one patch/update breaks other services/functionality?

Re:Say what?

[SuiteSisterMary]

Go home, plug into cable/DSL. Get infected. VPN in to Microsoft. Ooops.

That's the SP, not the patch!

[burgburgburg]

The service pack was released January 17, 2003, a week before Slammer hit. The patch, which does require all of that manual effort is what has been out there for the past six months.

Alot of sysadmins were waiting for the SP to be released before even approaching this one, just because the patching process is so complex. They just waited a week too long

Re:To clarify some Myths about Slammer

[delus10n0]

Ive read that the patch before this thing went big was a bitch. Basically it was a lot of manual this and that updating and rebooting. Basically this meant a lot of people couldnt get aproval from management to patch the server.

I don't understand where the difficulty comes in. You run the service pack, it extracts to your root drive (or whereever you want it to, actually.) You then run setup.bat -- That's it! You're patched!

Some have said they applied the patch and still were vunerable./Some have said the patch fucked their server.

It worked for me, and I can't speak of the ones it didn't work on. SP3 came out a few weeks ago, and admins should have at least had that installed.

Also, I think I read that the cumulitive SQL server patch that was supposed to be out a long time ago finally came out as soon as this worm hit.

Bullshit. The patch to fix the overflow problem came out half a year ago. And the service pack (which includes the patch) came out a few weeks ago.

Don't believe everything you hear..

Re:Grounds for legal action?

[horatio]

You make a very good point that I didn't see in the last round of comments when /. broke the story. There were, however, excessive numbers of comments along the lines of "those people are so stupid why didn't they just patch their systems".

Personally, I don't run M$ servers. When I did, it was a bitch. Patches that "fixed" one thing broke two other things, and that which was working properly stopped.

One specific example is in the NT service pack series: between SP3 and SP4 (I believe) the format/structure of the NT filesystem changed. If you formatted a partition in post SP3 (SP4 or greater), you could NOT access the partition when you had to rebuild the system (6-12 mo later) UNTIL you made it back up to SP4+.

Why do I want to apply a patch which will probably break my system? To close a security hole, yes. While I'm a bit of a security 'nazi', I wonder sometimes if the patch is worse than the risk of a security breach.

Re:Never patch a running system ;-)

[KjetilK]

It could be something in that, given that Unix has a large market share on the server side.

However, we should be very careful about bragging about it, because as it turns out UNIX admins are not that fast [rtfm.com] either. A study of an OpenSSL vulnerability and the subsequent release of the Slapper worm shows that many admins need some fire before they get moving.

My Experience

[0xA]

Ive read that the patch before this thing went big was a bitch. Basically it was a lot of manual this and that updating and rebooting. Basically this meant a lot of people couldnt get aproval from management to patch the server.

It's not that bad really, I think later versions of the patch even included a batch file to copy stuff around for you. Even without it, it only took 10 minutes... I mean really, if somebody can't handle this kind of stuff should they really be an admin?

Some have said they applied the patch and still were vunerable.

Yeah you have to be careful with this stuff, if you apply patches in the wrong order you can sometimesend up with the vulnerable code still there. I know a _really_ good admin that got hit with Code Red because of that. The correct order can sometimes be a bit of a mystery.

Some have said the patch fucked their server.

That's the big problem with this situation. I can understand why people don't have this patch or SP3 installed. You really never know what one of these things is going to do. It is common for amins to schedule a 3 hour downtime to roll something like this in, even if they have tested the hell out of it. You need time to get the damn thing back out if it screws stuff up. I deployed W2K SP3 onto my terminal servers a few months ago and it broke Office on every one of them. It didn't do that when I tested it, took me hours to clean it up.

Re: Which would you rather have?

[Black Parrot]

> > Would you rather have a system where you have to manually implement every patch, or would you rather have a system where you didn't have any choices which patches were implemented?

> The first choice would lead to a lot more work.

I find the application-specific security patches we do under Linux to be trivially easy:

1. Read the announcement to see whether it applies.
   
2. Click link to download the update (else use ncftptget).
   
3. Verify checksum.
   
4. Apply patch by typing one line in su window.
   
5. [Restart daemon, if needed.]

The only time it's tedious is when it's a kernel upgrade (rare) or when it requires downloading something big from a patch-dotted server.

Re:Problem is IPv4

[trybywrench]

actually what made this especially bad was UDP. Not many programs run on UDP ports almost always they are TCP. TCP has a VERY important feature and that is upon a non-ack'd window it throttles back the send rate. This is a way to get congestion feedback to a host and tell it to "settle down". The problem with UDP is there is no way to tell it to slow down. Also, the fact that the Internet is a "best effort" network means that no matter what the UDP incoming rate the routers will do their best to deliver the packets. This comes at the expense of all other traffic flows in the router, no way to get congestion feedback to the host means no way to limit the incoming rate. Even if the routers just dropped the packets that still increases CPU and RAM utilization and with the volume that was happening would still probably bring traffic throughput to a trickle.

Re:I wonder how long...

[Anonymous Coward]

I don't think there is much danger to home users. While MSDE is included with those products it is not part of the normal installation. You usually have to run a second setup to install it. Even in VS.Net, if you choose to install MSDE, it simply copies MSI package to your hard drive and informs you that you will have run the install.

Re:SQL Server

[cptgrudge]

Windows 98 and NT 4.x will be entering "Non-supported phase" after June this year, Windows 2K even earlier, March.

Even earlier?

Uh, I'm sure you meant to clarify that while mainstream NT4 Workstation support certainly is up on 30-Jun-2002, Win2k Pro is up on 31-Mar-2005. That's 3 years, 3 months later than NT4. And the extended goes until the same date in 2007. That's over 4 years away. Win2k isn't going anywhere.

Do you think you'll still be using (insert current Linux distro) in seven years? What were you using seven years ago, and is it what you use now?

Companies cannot support a particular product forever, simply because they created it. I, for one, am glad that Microsoft does this. It enables me to phase out old systems, that, while useful in their purpose, are simply not cost effective to keep running anymore. It is a convenient excuse to say, "Sorry, but Microsoft doesn't support it anymore, so we can't. We'll either need to get a new system or you can go without support." Seven years is quite a bit of time, in the relatively fast moving tech sector. I realize that there are a bunch of examples where people need support for a large, unwieldy system that cannot be easily upgraded. But that is simply the nature of the beast. No one ever said doing this was easy.

Of course there are some people that say, "I'm still using Red Hat 2.1, and you'll never make me change!"