Forum: The Yahoo Denial of Service

It's one of the larger news items of the day, but we've sorta avoided mentioning it here because it is really "just another Denial of Service Attack." But it's the biggest one ever. It took down Ya- 'we serve half a billion pages a day' -hoo. And they were taken down for several hours from a distributed DOS attack. What does this mean? I honestly don't know, but I figure you guys might have some opinions.

Can I sue you for negligence?

If your system is cracked, and then used to attack me, can I sue you for negligence? How else do we get companies to put proper practices in place?

Like IP spoofing, for example. IP spoofing would more or less come to a halt if ISPs, Universities, and corporations would put some simple filters into place, preventing packets with impossible source addresses from leaving their networks.

This distributed DOS stuff can be stopped only if *all* of the sites in the community engage in sound security practices.

I don't think so

As I do not see a link out to anywhere I will guess that this refers to a problem that started yesterday and propigated throughout many top level routers. The problem originated at Alter Net and it would appear as though they had a bad routing update - which propigated to glbx.net and effected many sites such as Yahoo!, CNN and a few others. This all depends on who your connected into - and where the routing packets are forced, but for many USWest !nterprise customers yesterday half or so of the internet was "down".

Re:Any suspects?

Wired claims in Routers Blamed for Yahoo Outage [wired.com] that it was not a DoS attack; rather, it was a misconfigured router at their ISP. Anonymous source 'n everything.

Motives...

I wonder how long (or if it has already happened) until an employee of an online business decides to improve the value of his stock options by taking out his company's top rival(s) for a couple of hours. There are times (say around December 15th for many merchants) when something like could be devestating.

D-O-S: Not just for script kiddies any more....

jf

Window Shopping Hordes

Yahoo was taken down by a major Denial of Service attack--this is true.

What's really scary isn't DoS attacks that are obvious, but ones which are indistinguishable from regular traffic.

Reasonably static and well hosted sites like Yahoo wouldn't be taken out, but the average E-Commerce site, with dynamically generated pages off a single-point-of-failure SQL Server architecture would be completely knocked out by what appeared to be nothing more than extremely heavy traffic.

Such an attack would require massive compromise of hosts(since they'd be able to execute only a few five minute random clicksessions per hour), but would show up on no security scans and would be indistinguishable from an unusually large horde of window shoppers.

How would you defend against this? How would you even know you were under attack?

And, most intriguingly, if you're getting paid by the ad impression, would you care?

A quick message to the people responsible...your behavior will eventually lead to the kind of IP network monitoring that the Russian Government is making all their ISPs pay for. It is one thing to describe the attacks and work to repair the infrastructure; it's something entirely different to execute attacks that will quickly lead to solutions that can only be described as nightmarish.

Think for a moment who <i>wins</i> when you take down Yahoo, and shudder. Because there is a winner, and in the long run, it ain't you. You're helping someone. Guess who.

Yours Truly,

Dan Kaminsky
DoxPara Research
http://www.doxpara.com

Some relevant URLs on DDoS

1) stacheldraht" [sans.org]
2) trinoo [washington.edu]
3) tfn tribe flood network [washington.edu]
4) tfn2k [securify.com]
5) Cert's denial of service tools [cert.org]
Useful?

It means everyone...

...has to pay more attention to security. While I am sure there are quite a few people willing to cooperate in launching a DoS attack (and, BTW, who cares if it is typed DoS or DOS?), I'm equally sure the primary method is to launch the attack from the cover of a number of compromised systems. A DoS attack can be done with any compromised account, too. It doesn't require a "root" compromise if all you are doing is flooding a router or set of routers from multiple different networks. You only need a root compromise to do "cool" stuff with forged headers and illogical option bits (like SYN-FIN). If you are launching your attack from compromised accounts that you logged into from other compromised accounts, you don't care about forging headers. Your identity is already hidden by other means. What do you care if some suits come knocking on the door of the owner of the compromised host? You aren't there.

This means that we all have to take security seriously. That password matters. Don't share it. If you have resources, use two part authentication. Take reasonable precautions. Audit your setuid programs. Don't put "." in your path. Don't have world-writable files. If you can't afford commercial 2-part auth solutions, at least use ssh instead of telnet. Etc., etc., etc.

We can't afford to have security be the province of experts and miscreants. Responsible netizenship demands that we take security seriously, at least to enough of an extent that we can be confident our own systems aren't being used by others to attack systems.

Some people believe that cracking systems or launching DoS attacks are a legitimate form of civil disobedience. I actually agree with that. But you are only engaging in legitimate civil disobience if you are doing it on your own equipment and not concealing your identity. Protesters go somewhere openly and risk arrest. Vandals sneak around in the dark wearking ski masks and painting slogans. One is a principled stand and the other is a cowardly crime. Furthermore, when you use someone else's computer in your act of civil disobedience, it would be like the act of, when the police wade into your protest with their truncheons flailing, grabbing the nearest non-participant and using them as a shield. Cowardly.

So, as always shy with my opinions, that's what I think the giant DoS means.

Anyone know if this was mere mischeif or if there was a motive for this incident, BTW?

Re:What about prevention?

There are no defenses. Trust me, as someone who is deeply concerned about it and has spent a considerable amount of time investigating.

The attack doesn't attack your firewall, it doesn't attack your boxes, it very simply attacks your bandwidth, it fills it up, completely, leaving no room for other traffic.

It doesn't matter if your firewall drops every single packet it sees, for that matter it doesn't matter if you unplug your box, it isn't going to help at all.

The vast number of machines that have been compromised, especially on university campuses where attention to security is limited on many boxes, and a crack can go unnoticed for months or years, give these flood networks more bandwidth than a medium-large sized ISP. If they are willing to take the risk that someone tracks them down, they can knock out most companies and for that matter, often their upstream.

So, as an administrator, there is little you can do. Some things can help slightly, (see following) but if you get one of the larger networks pointed at you, you call your provider, get them to call their provider, and hope that they can implement some kind of filtering on their router as a temporary solution. You probably won't get far with that however.

Things to do:

1. log log log log log. Strange packets coming in should be logged. If you can do this, theres a chance the guy can be traced back to source if one of the IPs is on a network with a competent admin and the source of the network control packets can be found.

2. Alert whoever you have to. If you're getting hammered, its a crime, tell the police, look on the CERT site for more details about who you can contact if you're in this situation.

3. close up all ports that aren't critical, from any replies. These guys function best when they can hit a wide range of ports and get replies from your box, effectively doubling the load generated by each packet. If you drop 98% of the ports on your box, that leaves most of the packet hits out in the cold, making them have to work harder. Don't be scared to start dropping whole class A/B networks if a large number of hits are coming through from them.

4. For those using unix based firewall solutions, have a couple of scripts handy which you can use to turn off all ICMP (you should already be filtering bad ICMP, this just goes the next step), and all non-essential ports.

5. Have syncookies on your system if available, this will help keep you working during small TCP floods

6. Make sure that you, as admin, have on your firewall the necessary rules to deny spoofed IPs from within your own network. If you don't, you are irresponsible and quite possibly a contributing cause to this whole mess. An internet connected network needs monitoring, no matter how well set up. Take the time to do it.

The final verdict is there is no individual solution to this problem. If everyone implemented #6, we'd be in a lot better shape, still not brilliant but certainly a vast improvement. On the positive side, there are many brilliant minds who have observed this problem and are working on infrastructure solutions (see BOF recently etc).

No matter how good your firewall software, script kids these days have the capability to flood your entire link. Proactive and constant vigilance is the only thing that could possibly minimise the damage.

Links

Here's some links since none were posted:

Cyberattack Cripples Yahoo [apbnews.com] (APBNews)
Who's Behing Yahoo Attack? [zdnet.com] (ZDNet)
FBI talks with Yahoo! about attack [zdnet.com] (ZDNet)
How a basic attack crippled Yahoo [cnet.com] (CNet) (with stupid protocol animations too!)

And in other news: A different type of DoS attack is being carried out against Yahoo. At least 40 web articles [excite.com] have been written so far, showing evidence of how many reporters must be calling Yahoo right now. Once the second round of DoS attacks are stopped, the techies can finally get some work done beefing up the site.

Re:Packet Monkeys

I said it in my earlier post, but I'm going to say it again here (so, yes, mark me redundant if you must): Certainly a DoS attack can be a legitimate form of civil diobedience, but if you are going to do it as such, have the courage of your convictions and launch the attack directly from your own machines on your own network, using your real IP address. Then its civil disobedience.

My attitude towards Greenpeace protests would be quite a bit different if they went down to local nursing home, yanked old people out of their beds (they're easier to handle than say, rading a gymnasium), and chained them to the gates of a nuclear power plant.

When you sneak through other people's accounts, machines, and networks to both hide your identity and launch your attack, then you are effectively chaining up the elderly (metaphorically speaking, of course). For an act of civil disobedience to be an honourable act, one must openly reveal one's identity and run the risk of arrest and imprisionment. I'm not impressed if someone comes up to me and says "I told my girldfriend to chain herself to the gate. I stayed home. I had the sniffles."

Civil disobedience by proxy is the act of a coward. A sniveling little spineless coward.

My account info has my real name and my real primary e-mail address. I stand up for what I say. I don't lay booby-traps or hide behind other people.

Internet III and further

I wonder when we are going to start seeing subsets of the internet partition themselves off and only deal with other sites the implement certain policies (for example, contractual agreements regarding penalties from spam coming from your domain, failure to block impossible packets and so on).

It could be done pretty cheaply during the changeover to IPv6. Just use the first byte to indicate what level of security (or bitwise OR of different security features) the host network guarantees. Then you could just block, for example, any mail coming from someone who didn't guarantee they could track down the original author (whic implies that they have enforced similar rules on their relaying).

--Kevin

I'm sorry, I can't help it.....

When DOS attacks! This Sunday on FOX! (Right after the Simpsons!)

It is sweeps week after all....

Re:Can I sue you for negligence?

at first glance, fom an administrator's perspective, I can understand this.

however, once you take into account the realities of the machines that are on the net today, this is nigh impossible. every day, DSL and cable modems are bringing more and more windows, linux, xBSD, etc. boxes onto the net with assigned IP addresses and security holes the size of Texas.

you can't, however, pin this on these individual users. if you're a systems administrator and that's your only working task, it is still difficult to keep up with security issues these days. it's more than a full-time job to keep a network secure from all of the possible attacks. you're never going to get all of the broadband users to secure their systems themselves, it'd be a herculean task.

it's better to start at the software/OS distributors and force them to hande the situation better. much like setting up ipmasq for the first time, the first thing to do is deny everything, then allow only what is necessary. operating systems should install the same way.

jimmy installs redhat, and decides that he needs web, email, ftp and nntp access. he runs through the installation, and at the end only ports 80, 25, 21 and 119 are open. he doesn't know any more than that, and he shouldn't need to know more than that.

there's no bind running errantly on his system, no apache running... honestly, at the end of pretty much any linux installation users have daemons running that they'll never need or use, opening up ports and holes that just aren't necessary.

instead of expecting every single end user out there to attend BOF security conferences and read bugtraq, maybe we should give them more secure setups to start with.

after all, in your scenario BOF don't exist, since everyone would already be included.

Re:It's DoS

I'd guess that this is the work stream.c, a ip stack bug which panics/freezes(resource wise) and is not FreeBSD specific. One of the original bugtraq post actually included a Linux kernel panic line from dmesg. Reports were also sent in that NT servers were down aswell. Stream works by creating as many open files/sockets as the system will allow thereby rendering it useless and from what I've read that its effectivness is proportional to the volume of packets sent so modifying the standard distributed dos tool to send stream packets and therefore downed yahoo. Chances are the only reason yahoo got attacked was because it was there not because it was the only large network that had that hole in it network stack.
-Nick Chernyy
P.S. for all of you paranoid FreeBSD users, there is a patch available and has been merged into the sources long ago.

Re:It's DoS

No, it was in fact a distributed DOS attack.

Didn't you hear. It was caused by a bunch of DOS zealots who refuse to upgrade to Windows. They actually used DOS and just pinged the heck out of Yahoo. They claimed to be using this action as a way to show their disatisfaction with MS because they no longer support DOS. I, for one, say more power to 'em! Down with MS! Long live DOS! The undisputed KING of OS's!

----------------

"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein

It's a good thing Yahoo uses FreeBSD.

Recent tests with an exploit called "stream.c" -- which creates the same sort of denial of service situation -- showed that some Linux servers crashed when confronted with so large a flood of packets. But FreeBSD, while it did slow down, did not crash -- even if stream.c was tuned to cause the maximum possible amount of havoc on the network.

FreeBSD also has two special kernel options -- ICMP bandwidth limiting and TCP/IP RST restriction -- which can help with some DoS attacks. (No OS can do anything about a swamped pipe, of course, but if it knows how to throw away bogus packets and does not fall into the trap of trying to respond to them all, it'll be in much better shape. And, of course, it should never crash.)

I've seen some trolls in this discussion that suggested that FreeBSD was somehow responsible for Yahoo's woes. In fact, the opposite is true. If I'm going to get hit by TFN or Stacheldraht, I'll want a FreeBSD system -- probably the most recent version on the FreeBSD-stable development branch -- not NT, MacOS, or Linux. In our tests -- and we did a bunch of them when stream.c hit the streets -- it held up the best.

--Brett Glass

Re:Disabling mail forwarding

If all you want to do is to allow outgoing mail, just stop running sendmail in daemon mode. With redhat you can do this with

chkconfig sendmail off

As someone else pointed out, you also need to put a script that does ``/usr/lib/sendmail -q'' into /etc/cron.hourly/ if you don't want your mail to get stuck at random.

But another useful trick, if there are certain machines you want to accept mail from and others that you don't, is to run sendmail under tcpd so that it obeys /etc/hosts.allow and /etc/hosts.deny, by adding this to /etc/inetd.conf:

smtp stream tcp nowait root /usr/sbin/tcpd /usr/lib/sendmail -bs

That way you can, for example, let specific machines on your subnet connect to your SMTP port without allowing the whole world to exploit the sendmail-bug-du-jour. (You can also do this with ipfwadm firewall rules, but I find hosts.allow to be easier to deal with.)

I generally prefer running services on my desktop machines (including sendmail and httpd) from inetd instead of having them always running as daemons in the background because that makes it easier to centralize control of their access lists, and because you don't have as many idle processes chewing up swap space. And since I'm the only one who ever connects to the http server on my desktop machine, the process-creation overhead is trivial (this wouldn't be such a good idea for a high volume web or mail server, obviously.)

The big corporations can afford to write their own

The person you're hurting most, when you use the GPL, is the little guy -- the programmer who does not have the bucks to hire programmers to reimplement the code. This was Stallman's intent: to destroy programmers' prospects for success. He has said so, repeatedly.

Advocates of the GPL tend to invoke the bogeyman of large, evil corporations just spoiling to use your code. But if you buy this argument, you'll in fact be hurting the little guy who might challenge the big ones.

It's unethical to participate in an agenda whose purpose is to hurt others -- especially out of spite. Therefore, you should not use the GPL.

--Brett Glass

Re:Links

More relevant links that have emerged:

• Yahoo isn't the only site to be DoS'd out of existance: Buy.com, eBay and stamps.com have fallen as a result as well. Relatively intelligent report on MSNBC, but it seems to promote the idea that these DoS's are by the same group of persons (not that it's not likely, but I don't like the assumption). [msnbc.com]
  
• Informed article on Wired which asks the question "Was Yahoo Smurfed or Trinooed?" [wired.com]
  
• Anonymous engineer blames misconfigured router for Yahoo's troubles [wired.com].

---------
Question: How do I leverage the power of the internet?