Worst Security Clean-Up You've Performed?

nakhla writes "Last night, I was tasked (by my wife) to help fix her friend's computer. It is a Windows XP home system which has been running slowly, almost to the point of un-usability (like *that's* never happened before). It turns out that hundreds of random processes had filled up its meager 256 MB of RAM. The cause? Nearly 7,500 viruses and worms that had infected the system. That number doesn't even include the hundreds of spyware and adware programs that had installed themselves, as well. Although the box is now behind a firewall, that wasn't always the case. This was, by far, the most infected system I'd ever seen, but I'm sure it can't be the worst ever. What was the worst security cleanup you ever had to perform?"

Here's my toolkit...

[Anonymous Coward]

NB: posting as AC to prevent whoring

I've been working in the small shop/repair business for over 5 years, and its a weekly experience to get a machine in with thousands of trojans, viruses and spy apps. In cases where a re-install may not be desirable or feasable, here's a list of the tools we use to find, isolate and eradicate hostile software.

Disclaimer: I do not work for any of these companies, nor am I been paid anything by them. I just find that these tools work. Your mileage may vary.

1: Antivirus

As most of our customers are home users, we can recomend Grisoft's AVG as the most capable and reasonably priced ':)' antivirus out there. It does a pretty good job, and the installers are kept up to date so you don't have to fudge around with d'loading on a broken box.

AVG Free [grisoft.com]

2: Anti-Spyware

No-brainer. The best two in the business. Spybot and Ad-Aware. They don't get everything, but they both do a darn good job, and can even set themselves up to run on reboot before some of the uglies get going. We leave them on the system so we can attempt to train the user towards a safer future.

Ad-Aware Personal [lavasoftusa.com]
SpyBot S&D [safer-networking.org]

3: Process Viewers

Now this gets a little harder. Neither of these tools will do the job automatically, but with care, can show you the files and processes that are the center of these little problems. Personally, I like MS/Sysinternals Process Explorer, my boss prefers PrcView. As an interesting note: You'll occasionally find a hostile that can stop certain known process viewers from starting up. Get the old 95/98 version of PrcView. They always seem to miss that one. Recording the file name of the app, rebooting to the recovery console, and going in to hand delete the app works 98% of the time.

PrcView [teamcti.com]
Process Explorer [microsoft.com]

Now, the easy route....

Get yourself one of these. USB HDD Adapter Kit [tigerdirect.com] from your favourite retailer, and just hook the offending HDD up to a good machine with a up to date anti-virus scanner. You will have some broken startup and registry entries left over, but they're pretty simple in comparision.

I'd normally say, Enjoy! at this juncture. But you probably won't.

Best of Luck

kgs

Re: HOW did you clean it up?

[hcdejong]

My parents recently had a virus on their computer. No big deal (just one virus), but Norton AV couldn't remove it and the manual removal instructions Symantec gave were rather convoluted (Recovery console, blah blah blah). Solution: pull the disk, stick it in a USB box and hook it up to my laptop. Eureka! The disk is inert (it's no longer the startup disk), so you can repair at your leisure rather than trying to beat whatever got started up during boot. You have a functional system during the procedure (if for no other reason than to keep the removal instructions handy) and no arcana like the Recovery console. Also, you've got a virus scanner you know isn't compromised.

I know what I'll do next time.

Script Kiddie Hunt

[rwa2]

Back in college around 1998 my Redhat 5.x box got remote-rooted by some Samba exploit (the exploit was called ADMmountd). Most of the standard utilities like ls and top and ps were modified to not detect the rootkit, but du stopped working completely, and I managed to stumble upon the rootkit files in a hidden directory in /usr/lib/.lrk or something like that. Then I noticed IRC callback connections in tcpdump and followed the trail to some swedish IRC server. But didn't really get any leads there.

It was pretty good about cleaning up after its last logs, but I finally managed to stumble into the kiddie's home dir on my box... the damned kiddie forgot to clean up his .bash_history ! Well, actually he did (as evidenced by some rm ~/.bash_history commands in his .bash_history), but of course his shell wrote it from memory again on logout. I found some entries there that led me back to another server he compromised.

Looking at that (also Redhat 5.x) server's web site, I noticed that it had some evil users who exposed /etc/passwd in some cgi scripts. This was before Redhat started using /etc/shadow, so a few cycles of john-the-ripper later I had a list of remote login accounts and most of their fairly trivial passwords (including root). Probably the exact same way the script kiddie took over that box. So I sent an email to the admin of that server, and (as it was some other poor college bastard) surrendipitously logged in to /his/ rooted box, did some additional forensics. The home base apparently was at goethe.sbu.edu , which apparently hosted some bored-looking CS guy (there were only 7 enrolled in the program :P )at St. Bonaventure University, though he may as well have been rooted himself. and cleaned up the rootkit on the remote machine as well, shutting off the compromised services and accounts before leaving myself.

So I cleaned up some other computer as well as mine. That was pretty much the time I migrated to Debian for good... haven't had nary a problem before or since. ;>

Anyway, here are some annotated excerpts from the .bash_history I archived:
blksheep/.bash_history

cd /tmp
cd .ADM
ls
ADMmountd liuxcentral.com -t 0 # plenty of typos while "scanning" for vulnerable hosts
ADMmountd linuxcentral.com -t 0
ADMmountd www.mondenet.com -t 0

# retrieving the logfile cleaning utility, which didn't work on .bash_history, apparently
ftp goethe.sbu.edu
mv utclean.wri utclean.c
gcc utclean.c -o utlcean
mv utlcean
mv utlcean utclean
chmod +x utclean

# Testing his rootkit
who
ls
screen find / -name .wh00p -print >>blah
ls
cat blah
rm blah
cat /usr/bin/.wh00p .wh00p # I guess this was the real "who", he ran this often to watch his back, I suppose

Re:HOW did you clean it up?

[walt-sjc]

how do you guys get rid of these nasty rootkit and evolved spywares which can hide very well without reformatting

You don't. It is not worth the time and effort unless your personal / professional time has zero value. Get your data off and reinstall / restore from image.

Otherwise (if you are getting paid well for it) you can boot off a live CD or install the drive as a second in another system (one that has all the autorun crap disabled), Run AV/AS(pyware) on the drive, edit the registry removing all the startup items that you know isn't needed, run md5 comparisons on all the system files, and go from there. Dumping the registry and comparing with a known good registry is helpful at spotting crap.