Slashdot Log In
Akamai DNS Outage Messes up Net
Posted by
CmdrTaco
on Tue Jun 15, 2004 09:39 AM
from the point-of-failure dept.
from the point-of-failure dept.
katre writes "Checking all my favorite sites this morning, I saw that about half a dozen seem to be offline. Trying to figure out why, I found an interesting article on the front page at http://isc.incidents.org/. Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others. Whatever happened to my decentralized net with no single point of failure?"
This discussion has been archived.
No new comments can be posted.
Akamai DNS Outage Messes up Net
|
Log In/Create an Account
| Top
| 522 comments
(Spill at 50!) | Index Only
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
|
2
(1)
|
2
I'm definitely not a technical guru... (Score:5, Interesting)
(http://www.opaquelucidity.com/)
Re:I'm definitely not a technical guru... (Score:5, Insightful)
Re:I'm definitely not a technical guru... (Score:5, Informative)
(http://coolerthanyou.net/)
Re:I'm definitely not a technical guru... (Score:5, Interesting)
I realise that some jobs are much more inpractical when there is downtime, but not everybody even here on
Re:I'm definitely not a technical guru... (Score:5, Interesting)
(http://somethingstirring.blogspot.com/ | Last Journal: Wednesday November 14, @02:17PM)
Re:Uh (Score:5, Insightful)
(Last Journal: Thursday February 05 2004, @11:30PM)
It is misleading to refer to the box as a "Linux" box. Was it really the kernel that was at fault for the machine being cracked, or was it a bug in one of the daemons that the machine was running? There are differences between a Linux box that runs BIND and another that runs EZ-DNS (or whatever).
How about this: Instead of labelling the Akamai boxes that have problems as "Linux" boxes, label them as "BIND" boxes, or whatever DNS server it is that it runs. Perhaps there's a FreeBSD machine in there that is having similar problems.
It is allowable, though, to refer to a Windows box as just that. MS ships an all-in-one product, and seldomly do admins use Windows to run BIND, Apache or other OSS servers.
All of this hand-ringing in an effort to paint "Linux" as bad, or as "just as bad" is dopey. One might as well point a finger at the administrator of the machine that was hacked, the services that were running on it, etc. Most Windows problems are caused by the same thing too. It is wiser to point at the admin (and the services one chooses to run) than to point at the OS, or the kernel.
Re:Interesting... (Score:5, Funny)
(http://www.utimegames.com/)
(Apologies to whomever I'd seen that from before.)
Re:Interesting... (Score:4, Funny)
Re:Interesting... (Score:4, Informative)
You are also confusing their cache servers with their DNS servers. They're completely different.
Akamai does use *some* win servers (Score:4, Informative)
Re:I'm definitely not a technical guru... (Score:5, Insightful)
*Live* and *work* are too entirely different things. I could not get any of my work done with network access.
Re:I'm definitely not a technical guru... (Score:5, Funny)
Errr, obviously I mean without network access. Although I'd spend less time on Slashdot so perhaps I can't get my work done with network access.
Re:I'm definitely not a technical guru... (Score:5, Insightful)
(http://home.earthlink.net/~bluethundr | Last Journal: Tuesday August 19 2003, @12:23PM)
F'real. To think, they [barnesandnoble.com] did [barnesandnoble.com] all [barnesandnoble.com] that [barnesandnoble.com] even before the Altair was a twinkle in Ed Roberts' jockey shorts!
Re:I'm definitely not a technical guru... (Score:5, Funny)
(Last Journal: Monday November 08 2004, @03:43PM)
Offline working can be surprisingly productive
Because that means then you aren't on slashdot?
er....brb, I should probably get back to work.
Re:I'm definitely not a technical guru... (Score:5, Insightful)
(Last Journal: Tuesday September 13 2005, @03:45PM)
Anyways butting both DNS records on the same point of failure breaks standards. These companies deserve to be hit hard (PR wise) for not building a roburst network.
Re:I'm definitely not a technical guru... (Score:4, Insightful)
Re:I'm definitely not a technical guru... (Score:5, Insightful)
"DNS was not quite designed in such a way" (Score:5, Insightful)
(http://slashdot.org/~Ernesto%20Alvarez/pubkey | Last Journal: Thursday November 10 2005, @07:37PM)
DNS was designed to be robust enough. Not one root server but many (ok, that's the weak point, we've all seen many DDoS against them, but it's not THAT bad). All zones are handled by their own servers, and (in theory) multiple servers for each zone. All in all, it's not a bad design.
If what happened was that someone put all the servers behind one link, it's not DNS' fault, the BOFH there screwed up (and considering it's akamai, they should not have done that).
(If that's not what happened, sorry, I couldn't RTFA, it's slashdotted or there's some sort of DNS problem there too).
novell and dns... (Score:4, Insightful)
One of the neat things was the log screen that showed dns actions and you could follow the trail of dns requests to see how they were resolved. what makes this not O/T is that i beleive that this went into a log.
The reason that I think about that is, if DNS stopped working, i'm not sure that i have cached numbers that i could easily get to....
eric
Re:I'm definitely not a technical guru... (Score:5, Informative)
(http://www.macphersonclan.com/rod)
go to <a href="http://www.dnsstuff.com/">your favorite DNS lookup page</a> and lookup the akamai hosted site. (getting the real address rather than the akamized version) Now open your hosts file and add that in.
Now you will always get the non-akamized version of that site. Akamai problem solved.
I keep google in my hosts just so I can be sure that DNS issues like this won't cut me off from my favorite search engine.
Re:I'm definitely not a technical guru... (Score:4, Funny)
(http://bance.net/ | Last Journal: Sunday December 28 2003, @08:50PM)
add esignal too (Score:3, Insightful)
(http://www.rant.st/)
hope the al quedas aren't taking notes on this..
Well . . . (Score:5, Insightful)
(http://www.trustedworlds.net/)
Its still there, and you're using it. The only organizations affected by this are those who chose to use a service that acts as a single point of failure.
Root servers not decentralized? (Score:5, Insightful)
(http://ottodestruct.com/)
The root nameservers are the most obvious example...
The most obvious example? The fact is that there are 13 of them, in widely scattered locations across the globe, and it's not decentralized?
Damn man, what exactly would you consider "decentralized" then?
Root servers go down all the time. It's not particularly unusual. There's THIRTEEN of the things. Up to 8 have been down at once with no major effects on the network, IIRC.
Re:Root servers not decentralized? (Score:5, Interesting)
(http://syberghost.livejournal.com/)
Damn man, what exactly would you consider "decentralized" then?
Akamai has 13, in widely scattered locations, as well. That in itself doesn't make them sufficiently decentralized.
The reason the root servers don't have this problem is that they don't all run the same software (anymore) and aren't all administrated by the same people.
I'm making an assumption here, of course, but I will not be a bit surprised if it turns out that Akamai loaded something that hit all their routers at once.
Re:Root servers not decentralized? (Score:5, Interesting)
(http://www.omnifarious.org/~hopper/ | Last Journal: Tuesday October 02, @12:21PM)
The root nameservers are not under decentralized political control, which still makes them a single point of failure, albeit a different kind of failure.
Re:Root servers not decentralized? (Score:5, Insightful)
To be truly decentralized not only do we need more than 13 overloaded root servers, but no one entity should be authoritative. How that's done is left as an exercise to the reader.
Re:Root servers not decentralized? (Score:4, Insightful)
(http://www.tlarson.com/)
I'm sorry, my friend, but it most certainly does mean decentralized. Here's why:
Decentralized means "having power or function dispersed from a central to local authorities". Each individual top-level nameserver operates entirely independantly of the others to the extent that it is capable of remaining completely operational in the absence of the others.
DNS is actually the epitome of a decentralized service--as perfect an example as there comes. Assuming it is implemented as perscribed in the RFCs, there is no single point of failure (an incorrectly implemented DNS system is not the result of a poor design, it's the result of poor implementation--you can't blame DNS).
There are 13 totally and completely independant top level servers. The only thing that ties them together (in a practical sense) is that they speak the same protocol and synchronize with eachother if possible. All top-level domains have at least two nameservers (generally much more), and all second level domains are required to have at least two authoratative nameservers as well. If any one of these servers in the whole chain fails at any time, the others will pick up the slack--it's part of the protocol.
Implementing this service correctly such that no failure will take down your own domain is left as an exercise for you. It's your domain and your nameserver. You're responsible for insuring that it works. The "system" correctly assures that each one of your own nameservers will be queried until one responds. If you take all of your own nameservers offline, there's obviously nothing that the DNS system can do to help you. That's what Akamai's problem was. Don't blame DNS.
Re:Well . . . (Score:5, Insightful)
(http://www.traxel.com/)
Its still there, and you're using it. The only organizations affected by this are those who chose to use a service that acts as a single point of failure.
You said it brother (and beat me to the punch). This is a clear talking talking point for anyone who is attempting to justify avoiding a monoculture. When you brings up Microsoft, around which revolve a number of good examples of the dangers of monoculture, you risk the debate turning political and will almost certainly be discounted as a Linux/Apple/Unix zealot by at least some in the listening audience. It is very worthwhile to have other examples besides Microsoft and cotton when explaining the risks.
Whatever happened to my decentralized net? (Score:5, Funny)
Whatever happened to my decentralized net with no single point of failure?
Its there. Get out your old Usenet reader. See, you still have your porn.
points of failure (Score:5, Interesting)
(http://www.oculon.org/)
I think everyone has several "single" points of failure -- my cable modem dies at least twice a month and my wireless router conks out at least twice a day
Clear your cache (Score:5, Informative)
(http://www.frennzy.net/)
If you clear your cache, you will probably get the new entries, unless your ISP hasn't caught onto the problem yet.
Re:Clear your cache (Score:5, Informative)
(Last Journal: Sunday September 07 2003, @03:00AM)
ipconfig
Re:Clear your cache (Score:5, Informative)
(Last Journal: Wednesday October 15 2003, @05:16PM)
lookupd -flushcache
ok (Score:3, Funny)
Good morning, Mr. Gore. (Score:5, Funny)
How ya doin', Al?
Re:Good morning, Mr. Gore. (Score:4, Funny)
(http://members.cox.net/jmccorm)
Re: Good morning, Mr. Gore. (Score:5, Funny)
(Last Journal: Thursday October 19 2006, @09:26PM)
In Soviet Russia the world revolves around YOU!
"He's sick of the jokes boys. Let's shut 'em down." -- Chief Wiggum
Ironically... (Score:5, Informative)
Single point of failure (Score:5, Funny)
(http://www.elizondo-family.net/)
Re:decentralized DNS is a pipe dream (Score:4, Informative)
(http://alec.restontech.com/ | Last Journal: Monday March 06 2006, @12:54AM)
Hmmm (Score:5, Funny)
(Last Journal: Friday June 18 2004, @11:45AM)
Whatever happened to your decentralized net? (Score:5, Insightful)
(http://seankelly.biz/)
Too bad even the term P2P raises so many red flags with certain Associations of America.
Re:Whatever happened to your decentralized net? (Score:5, Informative)
(http://www.lawyers.com/)
They have a private cached network they sell access to. It's like taking a service road around crowded highways to get closer to the final destination.
One of the companies I used to work for used Akamai, nice network... not so great customer service unless you are a really big customer.
2nd time in a month (Score:5, Informative)
(http://www.zhadoom.com/)
Re:Terrorist attacks, anyone? (Score:5, Informative)
DNS issue... (Score:4, Insightful)
(http://www.tuxedojack.com/)
Re:DNS issue... (Score:5, Funny)
(http://obsessivemathsfreak.org/ | Last Journal: Friday June 09 2006, @08:15PM)
I'll be at 127.0.0.1 until this blows over.
releted to linux kernel DoS exploit? (Score:4, Interesting)
Re:releted to linux kernel DoS exploit? (Score:5, Funny)
(http://www.klaproos.net/)
Iam now trying to send the porn but the mail server is unreachable.
Yahoo (Score:3, Funny)
(http://www.loscreepers.net/)
my failure (Score:3, Funny)
My central point of failure...
Preformance vs reliability (Score:3, Interesting)
Lack of notification (Score:5, Interesting)
Are these guys so convinced of their omnipotence and indispensibility that they don't feel the need to communcate with the world about what is going on?
sPh
Re:Lack of notification (Score:4, Funny)
(http://syberghost.livejournal.com/)
Yeah, they should post a notice on their web page, saying their internet connection is down. Bastards.
I'm surprised... (Score:5, Funny)
(http://www.samandlynette.com/)
Well, it wasn't out for that long ... (Score:4, Informative)
(Last Journal: Tuesday May 04 2004, @09:18PM)
Pwned by CNAME to Akamai?
(You can't have CNAME records for the base domain, hence google.com would have had an A record instead, whilst www.google.com would have been a CNAME to akamai)
can we figure out... (Score:4, Funny)
(come on, it's funny. at least I didn't suggest blaming SCO...)
Akamai is evil! (Score:3, Insightful)
(http://scovetta.blogspot.com/)
Lack of multiple points of failure (Score:5, Insightful)
(Last Journal: Saturday November 06 2004, @08:56PM)
"Well, Akamai has a few million DNS boxes, if we put everything there we'll be fine! That's not a single point of failure!"
Yeah, about that... multiple vendors may have been a good idea in retrospect instead of just one monolithic provider.
Time to re-examine the definition of Single Point of Failure.
You know... (Score:5, Funny)
(Last Journal: Friday June 18 2004, @11:45AM)
Let's see so far today.. We had a report on Yahoo... They're down. A report to a virus linked to Symantec.. they are up and down. We always link to Google, they are having problems... wooo. Now we just need another patent from Microsoft to bring them down... which by my records shouldn't be too long.
Easy to answer (Score:3, Insightful)
(Last Journal: Friday October 24 2003, @12:44PM)
The way to solve it is get more companies out there who provide the same sevices, something not easy after the dot bust era when people dont want to take such risks.
Need my Xerox fix! (Score:5, Funny)
(http://www.sciencemusings.com/blog/)
Checking all my favorite sites this morning...
Microsoft, Xerox and FedEx are some of my favorite sites too! But due to the outage I'm stuck slumming it here on Slashdot...
We fixed it quick (Score:5, Funny)
(http://www.apreche.net/ | Last Journal: Tuesday November 08 2005, @11:17PM)
We also developed a new DNS protocol in the process. ESEDOIM: Extremely slow encrypted DNS over instant messenger. Who wants to write an RFC?
Re:We fixed it quick (Score:5, Funny)
(http://bigmoneyjim.com/content/blogcategory/24/46/ | Last Journal: Tuesday March 28 2006, @07:41PM)
Akamai (Score:3, Informative)
For those that were wondering why it would affect DNS; Akamai somehow tinkers with DNS and BGP to redirect content to their edge servers.
As for Akamai being outdated, it still seems to me that its a good idea for Yahoo and some of the high traffic sites on the net. Akamai has thousands of distributed servers colocated with ISPs and NAPs. And they do seem to absorb nasty bursts in traffic (ie Star Report) better than a centralized server farm. But for their own sake, they better hope to not have another repeat of todays events.
Akamai's DNS black magic (Score:5, Interesting)
(http://francis.uy.googlepages.com/ | Last Journal: Wednesday August 29, @09:40AM)
It's not like a092156fg.akamai.net is in Seattle and k1039665.akamai.net is in Saskatoon. Instead, all of *.akamai.net goes to whatever cluster is "closest" to the requesting IP (based on BGP, Colonel's Secret Recipe, etc)
So if Akamai's DNS gets screwed up, I would expect major weirdness. And as more sites join EdgeSuite (where you host your entire domain on Akamai's servers & DNS) the effect must magnify.Of course, I could be completely wrong. I'm not a routing god, just a guy who thinks Akamai is a cool hack.
NANOG Postings (Score:5, Informative)
(http://www.econotarian.org/ | Last Journal: Tuesday May 18 2004, @02:14PM)
From here neither www.google.com, nor www.apple.com work. Both seem to return CNAMES to akadns.net addresses (eg, www.google.akadns.net, www.apple.com.akadns.net), and from here all of the akadns.net servers listed in whois are failing to respond.
I wonder (Score:3, Interesting)
This would seem an obvious solution. You are allowed to have many nameservers you know...
Success considered harmful? (Score:4, Insightful)
(http://slashdot.org/ | Last Journal: Saturday November 18 2006, @08:52AM)
If a product or service, such as Akamai, does their job very well, everybody will want to use them. If everybody uses them, you create a single point-of-failure. Any design flaw in that product or service becomes a disaster, simply through volume. Does this mean a successful product or service can actually be a bad thing for people?
Other examples include just about anything from Microsoft, older versions of Sendmail and BIND (worm-of-the-week problem), and Firestone tires.
(I'm not trying to advocate communism, excessive government regulation, or anything like that. So fanatical libertarians, conspiracy theorists, etc., can put down the rant-o-matic flamethrowers.
Comments?
Correction (Score:4, Insightful)
(http://slashdot.org/)
Point of Failure (Score:3, Funny)
(http://thelifeofbryan.multiply.com/ | Last Journal: Tuesday February 20 2007, @12:20AM)
Here's the Answer (Score:3, Funny)
(http://douglas.min.net/essay/)
You didn't pay the rent.
I noticed this problem this morning and 1st thing (Score:3, Insightful)
I tried pinging google and I was getting a reply so my first thought was, there is something terribly wrong at verizon DSL. I must make the most of what fragmented connection I have now before its down all day and I'm stranded actually doing work.
Thats when I started opening every story on slashdot's homepage in different tabs and setting them all to threshold 3, threaded... Just incase.
Come to think of it, I'm going to change my slashdot bookmark from slashdot.org to 66.35.250.151 just incase of DNS failure.
Need my SlashCrack
Luckily it's 99.45% shit to begin with. (Score:5, Funny)
(http://slashdot.org/ | Last Journal: Monday October 29, @07:20AM)
Dogpile (Score:3, Interesting)
(http://inglorion.net/ | Last Journal: Thursday October 06 2005, @07:17AM)
I wonder if Google will now turn to fully manage all their assets themselves...
Tech details (Score:5, Informative)
(http://slashdot.org/ | Last Journal: Saturday November 18 2006, @08:52AM)
Since a great many big name sites use Akamai, this effectively made large parts of the Internet unreachable. The destination servers themselves were up, but clients were unable to turn names (like www.example.com) into network addresses (like 192.0.2.42).
As Akamai maintains dozens, if not hundreds, of DNS servers across the globe, it is extremely unlikely that this was due to a normal equipment failure or DoS attack. Some kind of internal system trouble is much more likely. Whether a deliberate attack, or an accident, is unknown to me at this time. It could just be an internal configuration change blew up in a really bad way. Sh*t happens.
I do not know if this was just an Akamai DNS problem, or if other Akamai services were also affected.
Due to the way Akamai is usually implemented, it happened that, in many cases, the second-level domain names (like example.com) worked, but subdomains (like www.example.com and mail.example.com) did not. This is because most organizations put in CNAME records (pointing to names in *.akadns.net) for the subdomains. You cannot use a CNAME record for a domain that has other records, though, so most domains still had traditional A records, on their own nameservers, at the second-level.
The following sites/organizations are known to use Akamai: Yahoo, Google, Microsoft, Altavista, FedEx, Xerox, Apple
Reminds me of a story (Score:5, Interesting)
Being a geek, I thought up a list of about 30 sites to ping, scattered across the US. (.govs and
I freaked out a bit when the mid-atlantic seaboard came up missing. I crossed my fingers hoping that it was just some idiot who'd accidently cut one of the main fibers (which it what it ended up being) and not that Washington DC was now a big hole in the ground.
From Akami's Page (Score:4, Informative)
(http://www.afrobattle.com/ | Last Journal: Monday April 22 2002, @12:06PM)
How Sites are Coming Back Online (Score:5, Informative)
(http://www.econotarian.org/ | Last Journal: Tuesday May 18 2004, @02:14PM)
Google pulled references for akamais dns servers a short period ago. they are presently serving their own dns requests.
Also:
People seem to be getting around this by changing their DNS entries.
E.g. www.yahoo.com always used to be a CNAME for www.yahoo.akadns.net. But
now:
# host www.yahoo.com
www.yahoo.com is an alias for www.dcn.yahoo.com.
www.dcn.yahoo.com has address 216.109.118.64
www.dcn.yahoo.com has address 216.109.118.65
www.dcn.yahoo.com has address 216.109.118.66
www.dcn.yahoo.com has address 216.109.118.67
www.dcn.yahoo.com has address 216.109.118.68
www.dcn.yahoo.com has address 216.109.118.69
www.dcn.yahoo.com has address 216.109.118.70
www.dcn.yahoo.com has address 216.109.118.71
www.dcn.yahoo.com has address 216.109.118.72
www.dcn.yahoo.com has address 216.109.118.73
www.dcn.yahoo.com has address 216.109.118.74
www.dcn.yahoo.com has address 216.109.118.75
Which is owned by Yahoo! (via HotJobs.com).
Whatever happened to my decentralized net... (Score:4, Insightful)
(http://www.angelfire...epublican/index.blog | Last Journal: Thursday July 27 2006, @12:00AM)
Outsourcing and consolidation.
LK
Hmm . . . (Score:3, Interesting)
(http://www.hardcorehackers.com/~npsimons/ | Last Journal: Friday November 09, @03:00PM)
Hmmm, corporate whore much? Slashdot, Debian and my own two sites seem to be working just fine. Maybe the sites you choose to visit just don't get the 'net and it's decentralized nature.
it's like your rights, you can sign them away (Score:3, Interesting)
(http://slashdot.org/ | Last Journal: Monday April 16 2007, @01:18PM)
as I understand it, akamai is a distributed content hosting/caching service that also does DNS server services. they put a blade in your local ISP under contract, and popular pages from their customers serve off the local akamai server cache. they handle the DNS for those sites as I understand. if their blade caches get fed evil data, you get evil data, and www.fartblossom.org may disappear.
you can kill DNS by screwing up your own router, too. lots of ways to kill a distributed service that requires everybody to cooperate on a common set of standards and parameters.
Created SPoF (Score:5, Interesting)
(http://www.silverglass.org/)
The problem is that those sites created their own single point of failure by all using Akamai for DNS. When Akamai DNS fails, sites that depend on it for their own DNS fail.
It used to be nearly impossible for this to happen. The original rules for DNS were that you had to have at least 2 nameservers for your domain, preferrably 3 or more, and they couldn't be on the same physical networks. With that rule having a single network go down rarely made any domain unresolvable (backbone networks whose outages could render dozens or hundreds of other networks unreachable being the exception). Maybe we should put the old nameserver-diversity rules back into place.
"Caught in a BIND" (Score:4, Informative)
(http://crashrecovery.org/)
Robert
Google down? (Score:3, Insightful)
(http://www.nerdgod.com/)
Whatever happened to my decentralized net with... (Score:3, Funny)
(http://s87365085.onlinehome.us/ | Last Journal: Tuesday October 28 2003, @04:22PM)
Washingtonpost.com says it was a denial of service (Score:3, Informative)