Akamai: How They Fought Recent DDoS Attacks

yootje writes "Infoworld is running an interesting article about Akamai and the DDoS attack that hit the network of Akamai Tuesday. According to this article one of the defenses of Akamai is the big diversity of their hardware: 'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.' So says Paul Vixie, architect of BIND and president of the ITC." Yootje points to another article on this subject as well, this one at Internetnews.com. Update: 07/07 19:38 GMT by T : Note that Vixie's quote here is actually presented out of context; he was commenting by way of contrast on the diversity of the root DNS servers, not Akamai's content-serving system.

Wow

"We wired a million dollars into the attackers' Swiss account."

That's shocking!

Trade-Off

The diversity of hardware and software may be an IT nightmare but I think this shows how effective it really is. Now all we need is a concise cost/benefit analysis.

Re:Trade-Off

Allow me to perform a concise analysis for you. Hmm... the benefits are that DDoS's have some trouble knocking you offline. What are the costs? Much higher IT costs. Also, the total number of holes in your security will be higher. Just keeping track of all windows security fixes is hard. Imagine doing that for windows, solaris, linux, osx, and bsd. On 100 different hardware setups. Some things are going to go unpatched. You're giving hackers / crackers more opportunities, not more problems.

Re:Trade-Off

If you are Akamai, your uptime isn't everything, it is the only thing.

In their case maintaining a hybrid infrastructure makes perfect sense.
Remote exploit in IOS? No problem, the Juniper/Extreme/Linux/OpenBSD router in failover config takes over while patching goes on.

And if you are maintaining a massive hybrid infrastructure like that you will likely have the people and processes to handle security issues/patches.

Diversity Doesn't Refer to Akamai at All

I RTFA, and it doesn't say that Akamai has a diversity of hardware at all, that was talking about BIND:

Paul Vixie, architect of BIND (Berkeley Internet Name Domain) and president of the Internet Systems Consortium, charged that Akamai's proprietary approach to DNS makes it a single point of failure. He added that the 13 DNS root servers, which weathered a vicious DDoS attack in 2002, are even more defensible today than they were back then. The root servers are resilient, Vixie said, because their operators embrace diversity. "We deliberately use different operating systems, different name server implementations," etc...

AFAIK, all of the text that the quote from the submitter is regarding not Akamai, but BIND in criticism of Akamai. He's saying that they would have performed better had they used a more diversified network.

Correct me if I'm wrong.

Re:Diversity Doesn't Refer to Akamai at All

AFAIK, all of the text that the quote from the submitter is regarding not Akamai, but BIND in criticism of Akamai. He's saying that they would have performed better had they used a more diversified network

Paul should shut up about this topic. Companies should not go commenting about attacks made against their competitors - period.

His statement about the root servers is way off base. Only four of the 13 servers stayed up and the software running on them did not affect the outcome in any way. Most of the servers that went down were running a version of BIND as were two of the servers that stayed up. The other two roots were running ATLAS which is the ultimate in closed source proprietary systems, nobody outside VeriSign has seen the executable, let alone the source code.

I don't see how anyone could draw any conclusions either way on the basis of this sample. The distinguishing feature was the bandwidth available to the systems, not the software they run.

Paul should think more and speak to journalists less.

Re:Trade-Off

However, you are preventing your entire infrastructure to being nailed by a single exploit. With a monoculture, a single flaw exploited by a worm can destroy pretty much everything. With a mixed setup, although you have more possible entrances, each one allows a lot less damage.

If I have 1,000 troops, if I keep them all in the same fort, they will be a formidable force, unless I find the right weapon (like a nuke). If I keep them in 10 different forts spready throughout the country, although each one of them is more vulnerable individually, I have eliminated the possibility of everything being wiped out in a single blow.

Re:Trade-Off

Even with our little network (2 T1s, several servers) we do the same thing. Different OS versions, Bind builds, even Apache implimentations. NS1 is dedicated on a slow but extremely robust dual cpu box, all other boxes have a primary task and act as a back up for other tasks. At this small level, its not THAT hard to do, although it takes some preplanning and maintenance. Even the outbound linux router has an offline spare with a different version of Linux and completely different firewall/NAT configuration in case the first gets taken down.

IMHO, when it comes to providing IT services, if you are not paranoid, you are crazy.

Re:Trade-Off

Now all we need is a concise cost/benefit analysis.

Life versus death?

What you want out of backups and backup systems isn't so much that they are as good as or better than the primary systems, but that they are as independent as possible. Backing up OpenBSD to Windows 95 is not as stupid as it looks.

Re:Trade-Off

Akmai doesn't have a heterogeneous IT solution. It is the root nameservers that do. In fact, TFA says that the cost would be too high for them to do this.

Mod this whole story down "-1 incorrect".

Re:Trade-Off

So, in this case, not only did the submitter not read the article, but neither did the editors. I actually read the article and it was blatanly clear the the whole heterogeneous argument was *not* in reference to Akamai.

I just have one question: what exactly do the slashdot editors do? I thought they were there to screen incoming submissions. But obviously they don't. Basically, if that's their only job, they suck at it.

Sys admins

'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.'

Wow, your sys admins and help desk must LOVE supporting that!

Re:Sys admins

different operating systesm ... Wow, your sys admins and help desk must LOVE supporting that!

I know you were trying to be sarcastic, but I bet that they indeed do prefer things this way.

When the pager goes off at 3AM that there's a suspected new worm attacking your dos-based systems, it's nice to simply turn them off and let the other systems handle the load until morning when you can investigate the problem at your leisure.

Re:Sys admins

Can I ask an obvious question here?

Who the atech-ee-double-hockey-sticks runs "dos-based" systems anymore? I thought Microsoft abandoned the technology starting in 1995, and I personally submitted the "official end of life for DOS support" article to Slashdot several years ago.

We run heterogenious systems and support them because they provide different benefits and features for our many needs. Sometimes Windows OS servers actually are cheaper, more stable, and easier to support than their Unix counterparts. Sometimes not.

For instance, we have WebSphere running on Solaris and AIX as an app server platform, and it is great for high volume and failover. But we spend far more time (proportionally) troubleshooting that technology (and the hundred or so servers that run it) than the .NET application servers running on Windows 2000. As an app environment .NET is stable and actually quite fast, and run on much less expensive equipment. However there are only four of them and failover between boxes is sketchy, so on the rare occasion that there is a non-code related outage, it takes longer to get the environment back up to spec.

Just my anecdotal experience.

Wow...

They've achieved deliberately what happens naturally in a lot of other companies.

WRONG!

It says the root servers use different stuff, not akamai. RTFA.

Re:WRONG!

Exactly! Correct quotes from the article:

• Paul Vixie, architect of BIND (Berkeley Internet Name Domain) and president of the Internet Systems Consortium, charged that Akamai's proprietary approach to DNS makes it a single point of failure. ... [I]f Akamai tried to diversify the implementation of its large-scale content-delivery network, Vixie said, the cost would "drive their accountants crazy."

security by obscurity..

nobody knows what they run, so nobody can make a decent attack ..

Re:security by obscurity..

Sort of. You can know what they run, you can know you can exploit server A because it has a known vulnerability.

But servers B, C, D, E, F, G, etc are immune to your attacks on server A. To take down the root servers, you'd need to simultaneosly come up with 12 different exploits to knock each one of them out. Which makes it 12 times more difficult.

It's more proof of what I've always said, there is no "perfectly secure" OS in existence.

or...

couldn't you just link to them on slash dot
that's been proven to be an effective, system independent DoS attack (even if the attack was unintentional or brought about by the owner)

Re:Security through Stupidity

"Hey I'm in!"
ver^M
MS DOS 6.22
"wtf?"

Quote misattributed

Unfortunately, the ""We deliberately use different operating systems, different name server implementations..." quote is from Paul Vixie, president of the Internet Systems Consortium, and it's about the root name servers, not about Akamai.

Re:Quote misattributed

> Quote misattribute

Exactly. And Vixie goes on to say that Akamai can't do that because "the cost would 'drive their accountants crazy.'".

But I'm not sure having diverse bits of gear is such a huge cost. Wouldn't it instead be a way for sysadmins to broaden their experience and learn more about which tools are best for which jobs?

Re:Quote misattributed

The workplace is not a classroom, nor should it be treated as such.

If you have not realized that every place is a classroom, then, my friend, you have not learned a single thing.

Lack of diversity

If I read it right, one of their problems was their lack of diversity -- they all use Akamai's proprietary DNS.

Re:Lack of diversity

Also, Paul Vixie is the founder of ISC, not ITC. What a shoddy article write-up -- two blatantly obvious mistakes I caught by skimming the articles got front-paged.

intentional or not

I have to wonder if the diversity of systems was an intentional choice of theirs way back to face these kinds of attacks or if it just grew that way from rapid growth and having their systems spread all over.

They survived the attack and "Oh yea, we MEANT for it to happen that way".

I think it's spin.

Speeking of...

I don't know how related these two things are, but the AfterNET IRC network has been ^H^H^H^H^H^H^H is being flooded with SYN packets and is -down-.

Is this related to these DDoS attacks?

They never mention percentage of users impacted

Akamai claims over 1,100 customers and indicated that only 2 percent of them were noticeably impacted by the attack, such as not being available for about an hour.
Theo only statistic they ofer is the percentage of customers that were impacted. To me this hints of trying to play down the severity of the situation. When only 2 percent of your customers comprise (following is is a made up statistic since they didn't give me one) 80 percent of your traffic, you're lying by omission by only giving customer statistics.

The submitter is WRONG.

The submitter's description of the article was completely incorrect and backwards.

Diversity of hardware makes ROOT DNS SERVERS more defensible. Akamai is NOT diverse, and they do not want to be.

Submitters and Editors, RTFA!

The quote on diversity is by Vixie wrt the roots servers--it's a criticism of Akamai! Jesus H. Christ, it's in the first paragraph!

MacOS classic?

I've often wondered how a Mac running Classic on a beefy box as a server would stand up to an attemp to h4x0r it. To really get at it, seems to me you would have to get to the base underpinnings of the OS on some level. Which are arcane and hard to master, even (I'm told) to seasoned Mac programmers.

Not that I'm implying that it would be invulnerable to some attacks (like DDOS) but surely it seems that many of your other bases would be covered.

Re:MacOS classic?

I remember reading an article about the US Army using classic Mac for their webservers for just that reason. Hey, an URL: http://www.wired.com/news/politics/0,1283,21725,00 .html [wired.com]

This is an ad!

This article has nothing to do with Akamai, other than pointing out that Akamai DNS is vulnerable to DOS.

Most of this "article" is a puff-piece (or paid advert) for one "CloudShield Technologies," pimping their (vaporware) "server for applications that do deep packet processing at gigabit-per-second rates."

-Isaac

Authors should try readin the article

According to this article one of the defenses of Akamai is the big diversity of their hardware: 'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.' So says Paul Vixie, architect of BIND and president of the ITC.

Actually, according to the article the diversity approach is part of what's used to defend the DNS root servers, not Akamai. Vixie specifically mentions that this approach is not practical for an ordinary content provider like Akamai because, 'the cost would "drive their accountants crazy."' I'm dubious about just how helpful diversity would be against a DDoS attack in the first place. Diversity won't solve the problem of requests coming in faster than they can be processed.

Bad Link?

I love how the "ITC" links to www.isc.org

Uh, poster got it wrong

According to this article one of the defenses of Akamai is the big diversity of their hardware...

Erm, I think the poster made a mistake here. This diversity is attributed to the 13 root servers. Akamai's services do not employ such techniques due to the unsupportable cost. Based on the problems we saw during the DDoS, I can't say Akamai had much to offer in its arsenal.

Or am I the one who misread?

Different OS's?

Is that like using Windows 98 and Windows ME?

Akamai diversity?

Moderators, please correct the lead-in... BIND and the global DNS system is what has the diversity. The problem with Akamai was their lack of diversity on top of their proprietary hacks to DNS.

So what they're saying is...

'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.'

...That their entire operation is really based out of a bunch of Computer Renaissance stores and pawn shops run by cheap managers that don't talk to one another.

It sounds like a recipe for success!

Security through obscurity..

A valid tactic... it mitigates the problems with a unified vendor, but it costs lots more...

Gee-Wiz hardware will never win.

[description of magnificent gateway] For now the attackers are winning the arms race. The technology we'll need to monitor, react, and adapt in real time has yet to evolve, but it's headed in that direction.

I wish the net was headed in the right direction, but it's not. No single site or company will ever "win". The resilience of the web lies in it's redundancy and distribution. What I see is continued centralization and creation of points of failure. As "Broadband" internet access is more monopolized and treated as a platform for mindless browsing, and smaller ISPs are destroyed, the net is being squeezed into fewer and fewer hands. This invites attacks that can not be protected against. The real solution is to let everyone run everthing they want. That's the only way to route around damage.

Attacking Akamai with a DDoS...

...is like trying to wipe out swarm of gnats with a shotgun.

Re:Attacking Akamai with a DDoS...

So why did I go a few hours unable to get to Google a week ago?

Good old PR spin - nothing like it...

Boss: "Why did nearly half our service go down Friday?"

CTO: "Actually, sir, the real question is why did we lose less than half of our service. The answer is that I've, uh, been strategically using different systems and components throughout the enterprise on purpose to prevent drastic losses. No one else could have even kept 10% of their machines up under that DDOS."

Boss: "I knew I could count on you for the right PR spin job. Go back and think up some other good excuses."

-Adam

Ummm..

RTFA.

In the case of the Akamai incident, the vulnerable service was DNS. Paul Vixie, architect of BIND (Berkeley Internet Name Domain) and president of the Internet Systems Consortium, charged that Akamai's proprietary approach to DNS makes it a single point of failure. He added that the 13 DNS root servers, which weathered a vicious DDoS attack in 2002, are even more defensible today than they were back then. The root servers are resilient, Vixie said, because their operators embrace diversity. "We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures," Vixie told Internetnews.com.

He's not talking about how great Akamai is. He's talking about how great everyone else is.

On another note: What the heck does this story have to do with Akamai operators fighting DDoS attacks? They more than likely sat with their thumbs up their rears contemplating how having such a structured and inflexible DNS system could possibly be in err.

Interesting...

but why didn't it work? Or is this a case of "it could have been worse?" And if it is, then why does it even matter?

Windows

With all this diversity in system, one would think that setting up decoy Windows boxes would serve as good bait for hackers as well.

I was way off...

I thought we were disorganized here where I work, but it turns out we were just throwing up a good defense.

'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.'

RTFA first, please...

The article summary is incorrect. Diversity was not a defense for Akamai, it is a defense for the 13 DNS root servers. In fact, in the article, Paul Vixie "charged that Akamai's proprietary approach to DNS makes it a single point of failure." The diversity approach is what is used to help prevent these kinds of failures in the global DNS system.

Oooops

According to this article one of the defenses of Akamai

please reread the infoworld article, as they are refering to the DNS root servers, not akamai

Yootje Points?

What the heck are those? Are they like bad karma points for articles that have overlapping information with other articles?

By the way, which one of the articles is it that says Akamai did anything right to fight attacks?

extra secure systems

'We deliberately use different operating systems . . . .'

They called me crazy for using Windows 95, 98, 2000, CE and ME . . . I'm invincible! Bwahahaha!

Article isn't about the DDOS

'It's about CloudShield Technologies ... recently announced CS-2000', and nothing but a fluff peice meant to sell some hardware. Sure, Akami's DDOS is discussed ("DDOSs are ba-ad, mmkay."), but then it just goes on to talk about the CS-2000.

nobody read anything

not only did the submitter not rtfa

the editors did not rtfa

and after the first five posts pointing this out, it was obvious that nobody was reading the responses either.

nobody was reading anything, and now we have a 1000 responses saying the same thing, it wasn't akamai, it was the root servers, blah blah blah.

Fuck

I'm sorry, next time I will read the article ten times before I post...

Same cause as recent big electrical blackouts

I have a feeling it was more like,

(BOFH types RETURN, followed by)

"Oh Shit!"

Article is an ad for Vixie and his companies...

First, the root servers have different dns server software and OSes, not because Vixie thought of it, but because it is policy codified in the BCP RFC for root servers best practices [faqs.org]. In fact, I think he was unhappy about other root servers using non-BIND software in the beginning.

Second, he is being disingenuous about his comments about patents, his company owns at least one patent related to the Verisign "Site Finder" service methodology. Nominum Patent [uspto.gov] I didn't see any statements by him disparaging his company when they applied for that patent. So it isn't that he doesn't like patents, it is that he doesn't like that Akamai is making money doing third party DNS without paying him money or homage. Note: His commercial, for profit dns server software company has a white paper enumerating the scalability and other problems with BIND, and they use an architecture more similar to DJBDNS than to BIND 9 - separate auth and resolving dns server packages, most modern dns server software uses this architecture to reduce code complexity and improve security and performance.

Third, if he wanted to be the pillar of dns server software that he supposedly is, he could have sent a few goons from Nominum over to Akamai and set up some boxes with his commercial, for profit, "scalable" dns server software and Akamai would have been able to see if his software was able to stand up to the ddos attack better than what they have. If it did, he probably could have gotten a sweet, lucrative contract out of it and been a hero for helping thwart the attack, rather than a hypocritical, self serving competitor hiding behind Open Source to appear credible.

Fourth, Akamai is a single point of failure because that is what they do - offload dns and content load from the biggest companies on the net life MS, google and ebay. No, I don't work there, but I would venture a guess that they carry more traffic than (maybe) any other company. So I am sure it is easy to armchair quarterback and say they should do this and that, but when the attacks are probably at 10's or 100's of GiB/s I am not sure what I would do.

Nominum is also involved in RFID stuff, so I will be interested to see what happens with him and his companies as that ramps up. And who knows what deals have already been made - "the future of DNS is right."

Some DNS software links:
nsd - high performance, uses BIND style files and authoritative only [nlnetlabs.nl]
They have an interesting testing procedure where they run nsd and BIND, have them build responses to the same queries and then analyze any differences: diff analysis [nlnetlabs.nl]
maradns [maradns.org]
Powerdns, mysql and a pretty website [powerdns.com]
djbdns [cr.yp.to] he's grouchy and the no license license thing freaks people out and pisses them off, but people become attached to the quirky but rock solid software.
nstx, ip over dns, yeah... [sourceforge.net]

sikkerhetsfirma

new security company, fraud and ict sec.., I need help! wanna be my accomplice? Only open for the ict section, need folks who know nip, tcp/IP, ipSec, des/PGP, fourthfloor, nsk, an most urgent html security and programming.. check this out..
Helt seriost trenger jeg folk som snakker norsk ihvertfall, som kan deler av overnevnte, og/eller som har annen sikkerhets relatert bakgrunn innen hacker/cracker miljo.. dette er viktig for aa faa edge paa de andre etablerte firmaene som finnes allerede..

Give me a pip in [e-mail]

Og folkens, vaer seriose da.. trenger ikke crapmail!!!!!!

Re:I R 0wn j00

When you say "It didn7 w0rk" are you talking about the "Post Anonymously" checkbox?
Just askin you big hacker, you.

Re:Never heard of syn cookies or what?

SYN cookies are for TCP connections (because TCP uses a three-way
handshake to set up a connection). DNS uses (primarily) UDP traffic,
which is connectionless (there is no "stateful" connection with UDP).
SYN cookies do no good when your DNS servers are under attack.

Re:A stable version of BIND

IIRC, TinyDNS can integrate with LDAP, then you can SSH in and use an ldap browser/client to modify and add records..

It's a better solution, on paper, since LDAP is optimized for the fastest retrieval, at the expense of write time. RDBMS's are generally the other way around, or at least balanced.

Of course, you can have OpenLDAP use mysql as a backend if you really want to bring that abomination into the equation.

run Woody.

Bind9 in Woody never dies.

Re:A stable version of BIND

Any suggestions?

Yeah, stop doing whatever you're doing, and do something else. I've never had a problem with any version of bind on any operating system.

Re:A stable version of BIND

I ran BIND9 on Red Hat 7.2 for about 2 years.. its still running now in fact. No random crashes on BIND ever in that time. It was rock solid...

Re:wtf?

Not only that, Vint Cerf's name is Paul Vixie!

When will /. editors learn.

Re:What do they do?

For not knowing about the recent Akamai attacks, you must have just joined /. or been hiding in a cave for the past few months. Basically, a bunch of the recent worms that have been going around have a client built into them for targeted DOS attacks, and most of them target various servers in Akamai's network. For not knowing who Akamai is, you are just lazy. Try www.akamai.com. Akamai is a large hosting company (they estimate 15% of ALL internet traffic goes through them), hosting sites such as Microsoft. As for why the attack? Why does any site get attacked? Akamai is also a very large target, this attack just happened to disrupt service to 2% of its customers for a short time. And since you probably didnt RTFA, it was due to their DNS implementation. The rest of the article read like an ad for a new beast of a security server, and the article as a whole was rather uninformative and boring. The "Akamai got attacked" part was only in the first few lines.

tm

Re:A stable version of BIND

Do a search on freshmeat.net [freshmeat.net]. MyDNS [freshmeat.net] also runs straight off MySQL.

Re:Read the fucking article before submitting it

Dude, calm down. I'm sorry, I admit I wanted to have it fast on Slashdot, but not for my ego, but I like it to have it on Slashdot quick. You are talking to real persons, it was a mistake. Come on, it's not like your life depends on Slashdot.