Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
The Internet

Akamai DNS Outage Messes up Net 522

Posted by CmdrTaco from the point-of-failure dept.
katre writes "Checking all my favorite sites this morning, I saw that about half a dozen seem to be offline. Trying to figure out why, I found an interesting article on the front page at http://isc.incidents.org/. Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others. Whatever happened to my decentralized net with no single point of failure?"
This discussion has been archived. No new comments can be posted.

Akamai DNS Outage Messes up Net More

Akamai DNS Outage Messes up Net

Comments Filter:
  • but I believe the centralized concept of the 'net is something that is coming to an end, much to our loss. I'm pretty bothered by the fragility of this system. How many of you can't work without web access?

    • Re:I'm definitely not a technical guru... (Score:5, Insightful)

      by Malc ( 1751 ) on Tuesday June 15, 2004 @09:43AM (#9429644)
      How many *think* they can't live without web access? Offline working can be surprisingly productive, and as it often forces more thinking and planning (e.g. in preparation for being back online, and just thinking through what would happen of you could be online) the results end up being better.

      • Re:I'm definitely not a technical guru... (Score:5, Informative)

        by Pizzop ( 605441 ) on Tuesday June 15, 2004 @09:53AM (#9429786) Homepage
        It would be hard to do most of my work (Server Maint.) without the net. I might have to actually go to the servers instead of ssh. Wait, what am I talkin about, without the net I wouldn't HAVE a job.

        • Re:I'm definitely not a technical guru... (Score:5, Interesting)

          by Malc ( 1751 ) on Tuesday June 15, 2004 @10:16AM (#9430061)
          I'm not being condescending, but unless you're very junior or just a small pawn in a very large company, even server mainenance positions require some offline work. There's always some planning that needs to be done for tasks during today, the coming week, or even long term. Phone calls can be made, documentation updated, etc. It really depends on how long the outage is for and how quickly you're able or willing to switch gears and tasks.

          I realise that some jobs are much more inpractical when there is downtime, but not everybody even here on /. requires 100% connectivity. I myself live over 4,000 km from work (I haven't even met them face-to-face for more than four years) and I would be pretty pissed off and delayed on one of those days I have to do all my work via Terminal Services on machines at the main office or colocation facility... but let's be honest, a day's outage wouldn't mean I couldn't do any productive work. I once had a 10 day outage when the local telco switched my DSL line from interleaved channelisation to fast-path, but as I was in the middle of a long stretch of software development, I really didn't need a lot of internet. Batches of dialup and patience all-around sufficed ;)

      • Re:I'm definitely not a technical guru... (Score:5, Insightful)

        by AKnightCowboy ( 608632 ) on Tuesday June 15, 2004 @10:00AM (#9429867)
        How many *think* they can't live without web access?

        *Live* and *work* are too entirely different things. I could not get any of my work done with network access.

      • Re:I'm definitely not a technical guru... (Score:5, Insightful)

        by bluethundr ( 562578 ) * on Tuesday June 15, 2004 @10:00AM (#9429870) Homepage Journal
        ...how many *think* they can't live without web access? Offline working can be surprisingly productive, and as it often forces more thinking and planning (e.g. in preparation for being back online, and just thinking through what would happen of you could be online) the results end up being better.

        F'real. To think, they [barnesandnoble.com] did [barnesandnoble.com] all [barnesandnoble.com] that [barnesandnoble.com] even before the Altair was a twinkle in Ed Roberts' jockey shorts!

      • Re:I'm definitely not a technical guru... (Score:5, Funny)

        by endx7 ( 706884 ) on Tuesday June 15, 2004 @10:05AM (#9429927) Homepage Journal

        Offline working can be surprisingly productive

        Because that means then you aren't on slashdot?

        er....brb, I should probably get back to work.

    • You mean decentralized?
      Anyways butting both DNS records on the same point of failure breaks standards. These companies deserve to be hit hard (PR wise) for not building a roburst network.

    • Re:I'm definitely not a technical guru... (Score:4, Insightful)

      by jocknerd ( 29758 ) on Tuesday June 15, 2004 @09:43AM (#9429653)
      I actually would probably get work done without web access!
    • I do. In fact, I'm trying to move as much of my work onto the web as possible. First it was email, then my calendar and contacts (Horde). Now, I'm starting to run finances online so I can access it from anywhere. A few years ago there was a lot of hype about moving all software to webbased software and I think this is still the eventual outcome of the internet. It makes sense to everybody. No costs in making CDs or packaging, ability to work on anything from anywhere, and easy ability to upgrade softw

    • Re:I'm definitely not a technical guru... (Score:5, Insightful)

      by fish_in_the_c ( 577259 ) on Tuesday June 15, 2004 @09:53AM (#9429788)
      you can still get to all those sites. You just have to REMEMBER the ip instead of depending on the computer to look it up for you ;). TCP/IP was designed to have not centeral point of failure and still does it's job well. DNS was not quite designed in such a way.

      • "DNS was not quite designed in such a way" (Score:5, Insightful)

        by Ernesto Alvarez ( 750678 ) on Tuesday June 15, 2004 @10:27AM (#9430200) Homepage Journal

        you can still get to all those sites. You just have to REMEMBER the ip instead of depending on the computer to look it up for you ;). TCP/IP was designed to have not centeral point of failure and still does it's job well. DNS was not quite designed in such a way.


        DNS was designed to be robust enough. Not one root server but many (ok, that's the weak point, we've all seen many DDoS against them, but it's not THAT bad). All zones are handled by their own servers, and (in theory) multiple servers for each zone. All in all, it's not a bad design.

        If what happened was that someone put all the servers behind one link, it's not DNS' fault, the BOFH there screwed up (and considering it's akamai, they should not have done that).

        (If that's not what happened, sorry, I couldn't RTFA, it's slashdotted or there's some sort of DNS problem there too).

      • novell and dns... (Score:4, Insightful)

        by ecalkin ( 468811 ) on Tuesday June 15, 2004 @10:58AM (#9430621)
        This was years ago (3? 4)... I set up a novell server and setup dns on it as a forwarder and pointed workstations to my novell server for dns.
        One of the neat things was the log screen that showed dns actions and you could follow the trail of dns requests to see how they were resolved. what makes this not O/T is that i beleive that this went into a log.

        The reason that I think about that is, if DNS stopped working, i'm not sure that i have cached numbers that i could easily get to....

        eric

      • Unless the server that lives at IPaddress W.X.Y.Z only hosts 1 server, and that server has it's documents in the server root folder. Most webservers any more use virtual name services to map HTTP requests to the right "web server" and set of documents.

        My personal server runs 7 domains with 12 or 13 sites. Some have real docroot folders, some use the default "you aren't looking in the right place" set of docs. But using an IP address to access a web site probably won't work in these days of many servers

    • Re:I'm definitely not a technical guru... (Score:5, Informative)

      by aonaran ( 15651 ) on Tuesday June 15, 2004 @10:07AM (#9429960) Homepage
      Solution to akamai problems:
      go to <a href="http://www.dnsstuff.com/">your favorite DNS lookup page</a> and lookup the akamai hosted site. (getting the real address rather than the akamized version) Now open your hosts file and add that in.

      Now you will always get the non-akamized version of that site. Akamai problem solved.

      I keep google in my hosts just so I can be sure that DNS issues like this won't cut me off from my favorite search engine.

    • Re:I'm definitely not a technical guru... (Score:4, Funny)

      by Shalda ( 560388 ) on Tuesday June 15, 2004 @10:32AM (#9430272) Homepage Journal
      Without web access, I have nothing to do but work.

  • add esignal too (Score:3, Insightful)

    by Lawrence_Bird ( 67278 ) on Tuesday June 15, 2004 @09:41AM (#9429626) Homepage
    provider of real time market data...

    hope the al quedas aren't taking notes on this..

  • Well . . . (Score:5, Insightful)

    by Maradine ( 194191 ) * on Tuesday June 15, 2004 @09:41AM (#9429630) Homepage
    Whatever happened to my decentralized net with no single point of failure?


    Its still there, and you're using it. The only organizations affected by this are those who chose to use a service that acts as a single point of failure.

    • Re:Well . . . (Score:5, Insightful)

      by Bob9113 ( 14996 ) on Tuesday June 15, 2004 @10:08AM (#9429976) Homepage
      Whatever happened to my decentralized net with no single point of failure?

      Its still there, and you're using it. The only organizations affected by this are those who chose to use a service that acts as a single point of failure.

      You said it brother (and beat me to the punch). This is a clear talking talking point for anyone who is attempting to justify avoiding a monoculture. When you brings up Microsoft, around which revolve a number of good examples of the dangers of monoculture, you risk the debate turning political and will almost certainly be discounted as a Linux/Apple/Unix zealot by at least some in the listening audience. It is very worthwhile to have other examples besides Microsoft and cotton when explaining the risks.

  • Whatever happened to my decentralized net? (Score:5, Funny)

    by mattkime ( 8466 ) on Tuesday June 15, 2004 @09:41AM (#9429632)

    Whatever happened to my decentralized net with no single point of failure?

    Its there. Get out your old Usenet reader. See, you still have your porn.

  • points of failure (Score:5, Interesting)

    by rlthomps-1 ( 545290 ) on Tuesday June 15, 2004 @09:42AM (#9429635) Homepage
    DNS dying on you? Just throw it on the pile of other connection problems

    I think everyone has several "single" points of failure -- my cable modem dies at least twice a month and my wireless router conks out at least twice a day ;)

  • Clear your cache (Score:5, Informative)

    by Frennzy ( 730093 ) on Tuesday June 15, 2004 @09:42AM (#9429638) Homepage
    Yahoo is already resolving through scd instead of akamai. I didn't check any of the others.

    If you clear your cache, you will probably get the new entries, unless your ISP hasn't caught onto the problem yet.

  • ok (Score:3, Funny)

    by Anonymous Coward on Tuesday June 15, 2004 @09:42AM (#9429641)
    vague explanation, just a link to the ISC's Incidents website and not the article, and now that site is inaccessible courtesy the slashdot effect. Nice job, now we cant even find out what's going on!

  • Good morning, Mr. Gore. (Score:5, Funny)

    by Quarters ( 18322 ) on Tuesday June 15, 2004 @09:43AM (#9429645)
    Whatever happened to my decentralized net with no single point of failure?

    How ya doin', Al?

  • Ironically... (Score:5, Informative)

    by xbrownx ( 459399 ) on Tuesday June 15, 2004 @09:43AM (#9429646)
    ...I can't even get to http://isc.incidents.org/

  • Single point of failure (Score:5, Funny)

    by jelizondo ( 183861 ) * <jerry.elizondo@g ... m minus language> on Tuesday June 15, 2004 @09:43AM (#9429647)
    You could still access Slashdot, couldnt you?

  • Hmmm (Score:5, Funny)

    by Mz6 ( 741941 ) * on Tuesday June 15, 2004 @09:44AM (#9429666) Journal
    "Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and..."

    ... http://isc.incidents.org.

  • Whatever happened to your decentralized net? (Score:5, Insightful)

    by YetAnotherName ( 168064 ) on Tuesday June 15, 2004 @09:45AM (#9429674) Homepage
    The web happened my dear friend, and it was based on the predominant distributed computing model at the time: client/server. Even DNS, with its highly distributed spread of processing and data, has a set of (overloaded) root servers with the commensurate single points of failure. The solution? Peer-to-peer.

    Too bad even the term P2P raises so many red flags with certain Associations of America. :)

    • Re:Whatever happened to your decentralized net? (Score:5, Informative)

      by Tenareth ( 17013 ) on Tuesday June 15, 2004 @10:03AM (#9429904) Homepage
      Uhm, the root servers are not overloaded... this has nothing to do with the root servers, this has to do with Akamai having problems.

      They have a private cached network they sell access to. It's like taking a service road around crowded highways to get closer to the final destination.

      One of the companies I used to work for used Akamai, nice network... not so great customer service unless you are a really big customer.

  • 2nd time in a month (Score:5, Informative)

    by ZHaDoom ( 65485 ) on Tuesday June 15, 2004 @09:45AM (#9429684) Homepage
    This should cause some problems for akami, they had an outage may 24th. Once can be overlooked twice? these are some big companies they are going to be calling them. I bet there is some sweating techs in the cool noc [akamai.com] right now

  • DNS issue... (Score:4, Insightful)

    by Tuxedo Jack ( 648130 ) on Tuesday June 15, 2004 @09:46AM (#9429694) Homepage
    You would think that the root DNS servers would be kept up to date with critical information. Just what happened, and how did Akamai get knocked around this? Did they screw with their DNS information and change their nameserver addresses or something?

  • releted to linux kernel DoS exploit? (Score:4, Interesting)

    by Anonymous Coward on Tuesday June 15, 2004 @09:47AM (#9429702)
    Do we know if this at all related to the Linux kernel 2.4.2x/2.6 DoS exploit discovered yesterday?

  • Yahoo (Score:3, Funny)

    by blackmonday ( 607916 ) on Tuesday June 15, 2004 @09:47AM (#9429705) Homepage
    My Yahoo Email is down this morning, first time I can remember this happening. At least gotapex, techbargains and dealmac still work, otherwise I'd have to actually start working!

  • my failure (Score:3, Funny)

    by pcmanjon ( 735165 ) on Tuesday June 15, 2004 @09:47AM (#9429709)
    My primary point of failure is my router, the damn clip that keeps the cat6 cable plugged in the router always falls out.

    My central point of failure... :(

  • Preformance vs reliability (Score:3, Interesting)

    by Kardnal ( 471745 ) <{moc.liamg} {ta} {suitorg}> on Tuesday June 15, 2004 @09:47AM (#9429710)
    When Akamai's system was first announced, most people thought this was a great idea. It made sure that the sites that used this technology would always have the bandwidth they needed, when they needed it. Like with everything else in life, there's always a trade-off between preformance and reliability...

  • Lack of notification (Score:5, Interesting)

    by sphealey ( 2855 ) on Tuesday June 15, 2004 @09:48AM (#9429726)
    What ticks me off about this incidents (and I suspect that there have been several in the last 6 months) is that there is absolutely no notification given, either during or after the event. During this outage, some news outlets were still reachable (including Slashdot), and a simple notification would have saved hours (* 10s of thousands of network dudes worldwide) of time and much grief from the big bosses who couldn't reach Yahoo Finance, I mean critical business web sites.

    Are these guys so convinced of their omnipotence and indispensibility that they don't feel the need to communcate with the world about what is going on?

    sPh

    • Re: (Score:3, Interesting)

      by account_deleted ( 4530225 )
      Comment removed based on user account deletion

    • Re:Lack of notification (Score:4, Funny)

      by Syberghost ( 10557 ) <syberghost@@@syberghost...com> on Tuesday June 15, 2004 @10:15AM (#9430046)
      What ticks me off about this incidents (and I suspect that there have been several in the last 6 months) is that there is absolutely no notification given, either during or after the event. During this outage, some news outlets were still reachable (including Slashdot), and a simple notification would have saved hours (* 10s of thousands of network dudes worldwide) of time and much grief from the big bosses who couldn't reach Yahoo Finance, I mean critical business web sites.

      Yeah, they should post a notice on their web page, saying their internet connection is down. Bastards.

  • I'm surprised... (Score:5, Funny)

    by swasson ( 639367 ) on Tuesday June 15, 2004 @09:48AM (#9429731) Homepage
    that the /.'ers aren't trying to take credit for slashdotting the entire WWW.

  • Well, it wasn't out for that long ... (Score:4, Informative)

    by hattig ( 47930 ) on Tuesday June 15, 2004 @09:49AM (#9429739) Journal
    Typically, the domain itself (e.g. 'google.com') still resolves, but popular hostnames, like 'www.google.com' will not resolve.


    Pwned by CNAME to Akamai?

    (You can't have CNAME records for the base domain, hence google.com would have had an A record instead, whilst www.google.com would have been a CNAME to akamai)

  • can we figure out... (Score:4, Funny)

    by kaan ( 88626 ) on Tuesday June 15, 2004 @09:51AM (#9429758)
    ... a way to blame the outage on Microsoft instead of (or in addition to) Akamai?

    (come on, it's funny. at least I didn't suggest blaming SCO...)

  • Akamai is evil! (Score:3, Insightful)

    by scovetta ( 632629 ) on Tuesday June 15, 2004 @09:52AM (#9429766) Homepage
    When I was in grad school at Cornell [cornell.edu], my O/S professor went on a rant about the evils of Akamai. No one believed him. Now we know he was right.

  • Lack of multiple points of failure (Score:5, Insightful)

    by bastardadmin ( 660086 ) on Tuesday June 15, 2004 @09:52AM (#9429775) Journal
    I can see the logic that went into this plan:
    "Well, Akamai has a few million DNS boxes, if we put everything there we'll be fine! That's not a single point of failure!"
    Yeah, about that... multiple vendors may have been a good idea in retrospect instead of just one monolithic provider.
    Time to re-examine the definition of Single Point of Failure.

  • You know... (Score:5, Funny)

    by Mz6 ( 741941 ) * on Tuesday June 15, 2004 @09:53AM (#9429783) Journal
    If we timed our stories right we could pull this all off as the /. effect...

    Let's see so far today.. We had a report on Yahoo... They're down. A report to a virus linked to Symantec.. they are up and down. We always link to Google, they are having problems... wooo. Now we just need another patent from Microsoft to bring them down... which by my records shouldn't be too long.

  • Comment removed (Score:3, Insightful)

    by account_deleted ( 4530225 ) on Tuesday June 15, 2004 @09:57AM (#9429846)
    Comment removed based on user account deletion

  • Need my Xerox fix! (Score:5, Funny)

    by goober ( 120298 ) on Tuesday June 15, 2004 @10:04AM (#9429914)

    Checking all my favorite sites this morning...

    Microsoft, Xerox and FedEx are some of my favorite sites too! But due to the outage I'm stuck slumming it here on Slashdot...

  • We fixed it quick (Score:5, Funny)

    by Apreche ( 239272 ) on Tuesday June 15, 2004 @10:07AM (#9429966) Homepage Journal
    Yeah, google didn't work and we didn't know what to do. We tested and determined the problem was akamai within a minute. So I used AIM to ask a friend who could still resolve google what the ip was. he passed it to me over aim using gaim encryption no less. We then created an alias for google on our dns server. google.ourdomain.com.

    We also developed a new DNS protocol in the process. ESEDOIM: Extremely slow encrypted DNS over instant messenger. Who wants to write an RFC?

  • Akamai (Score:3, Informative)

    by junctionvin ( 128333 ) on Tuesday June 15, 2004 @10:08AM (#9429973)
    I run a small ISP and we happen to have 3 of their linux boxes on our network. I've never experienced a problem with them before today. For the hack of it we decided to just reboot their servers and now things are working correctly.

    For those that were wondering why it would affect DNS; Akamai somehow tinkers with DNS and BGP to redirect content to their edge servers.

    As for Akamai being outdated, it still seems to me that its a good idea for Yahoo and some of the high traffic sites on the net. Akamai has thousands of distributed servers colocated with ISPs and NAPs. And they do seem to absorb nasty bursts in traffic (ie Star Report) better than a centralized server farm. But for their own sake, they better hope to not have another repeat of todays events.

  • Akamai's DNS black magic (Score:5, Interesting)

    by frankie ( 91710 ) on Tuesday June 15, 2004 @10:09AM (#9429982) Journal
    Akamai uses (some would say ABuses) DNS in ways the rest of us (even global megacorps) wouldn't dare. Half of Akamai's magic is their 10000+ carefully-scattered servers, but the other half is their routing. Those servers are listed differently depending on where you ask from [washington.edu].

    It's not like a092156fg.akamai.net is in Seattle and k1039665.akamai.net is in Saskatoon. Instead, all of *.akamai.net goes to whatever cluster is "closest" to the requesting IP (based on BGP, Colonel's Secret Recipe, etc)

    So if Akamai's DNS gets screwed up, I would expect major weirdness. And as more sites join EdgeSuite (where you host your entire domain on Akamai's servers & DNS) the effect must magnify.

    Of course, I could be completely wrong. I'm not a routing god, just a guy who thinks Akamai is a cool hack.

  • NANOG Postings (Score:5, Informative)

    by TheSync ( 5291 ) on Tuesday June 15, 2004 @10:10AM (#9429996) Journal
    From NANOG [merit.edu]:

    From here neither www.google.com, nor www.apple.com work. Both seem to return CNAMES to akadns.net addresses (eg, www.google.akadns.net, www.apple.com.akadns.net), and from here all of the akadns.net servers listed in whois are failing to respond.

  • I wonder (Score:3, Interesting)

    by rabtech ( 223758 ) on Tuesday June 15, 2004 @10:11AM (#9430005) Homepage
    I wonder why these companies wholly switched their nameservers over? Why not have #1 and #2 be Akami, and #3 & #4 be your own nameservers? Preferably on different coasts or in different countries.

    This would seem an obvious solution. You are allowed to have many nameservers you know...

  • Success considered harmful? (Score:4, Insightful)

    by DragonHawk ( 21256 ) on Tuesday June 15, 2004 @10:12AM (#9430015) Homepage Journal
    I was thinking about this while scrambling to answer the phone, check outage reports, and generally calm down customers.

    If a product or service, such as Akamai, does their job very well, everybody will want to use them. If everybody uses them, you create a single point-of-failure. Any design flaw in that product or service becomes a disaster, simply through volume. Does this mean a successful product or service can actually be a bad thing for people?

    Other examples include just about anything from Microsoft, older versions of Sendmail and BIND (worm-of-the-week problem), and Firestone tires.

    (I'm not trying to advocate communism, excessive government regulation, or anything like that. So fanatical libertarians, conspiracy theorists, etc., can put down the rant-o-matic flamethrowers. :) )

    Comments?

  • Correction (Score:4, Insightful)

    by PhuCknuT ( 1703 ) on Tuesday June 15, 2004 @10:12AM (#9430017) Homepage
    Akamai didn't mess up the net. Akamai messed up some web sites that are akamai customers. Remember kids, www is only a subset of the internet, and akamai customers a small fraction of the www.
  • Judging by the response time of isc.incidents.org, I'd say slashdot is the single point of failure.

  • Here's the Answer (Score:3, Funny)

    by The Ape With No Name ( 213531 ) on Tuesday June 15, 2004 @10:17AM (#9430075) Homepage
    Whatever happened to my decentralized net with no single point of failure?

    You didn't pay the rent.

  • I noticed this problem this morning and 1st thing (Score:3, Insightful)

    by aardwolf204 ( 630780 ) on Tuesday June 15, 2004 @10:20AM (#9430106)
    I noticed this problem this morning when I was hunting for an updated version of YahooPOPs. I wasnt getting replies from Google. I opened another FirePanda window and my homepage, slashdot, was working fine (Hey look at that on the homepage, Yahoo changed their mail service today, no luck for YahooPOPs). I tried yahoo, altavista, even msn in different tabs but I wasnt getting anywhere.

    I tried pinging google and I was getting a reply so my first thought was, there is something terribly wrong at verizon DSL. I must make the most of what fragmented connection I have now before its down all day and I'm stranded actually doing work.

    Thats when I started opening every story on slashdot's homepage in different tabs and setting them all to threshold 3, threaded... Just incase.

    Come to think of it, I'm going to change my slashdot bookmark from slashdot.org to 66.35.250.151 just incase of DNS failure.

    Need my SlashCrack

  • Luckily it's 99.45% shit to begin with. (Score:5, Funny)

    by gelfling ( 6534 ) on Tuesday June 15, 2004 @10:20AM (#9430112) Homepage Journal
    Seriously we need a *.sht domain.

  • Dogpile (Score:3, Interesting)

    by RAMMS+EIN ( 578166 ) on Tuesday June 15, 2004 @10:22AM (#9430130) Homepage Journal
    So I wasn't the only one who couldn't get to Google the Great. Fortunately, Dogpile [dogpile.com] still worked. I used that meta search engine until Google started getting big and beating all the others in turning up relevant search results.

    I wonder if Google will now turn to fully manage all their assets themselves...

  • Tech details (Score:5, Informative)

    by DragonHawk ( 21256 ) on Tuesday June 15, 2004 @10:22AM (#9430133) Homepage Journal
    It appears that, at around 8:30 AM EDT (US Eastern Daylight Time), Akamai's DNS network experiened some kind of major failure. All of their DNS servers (that anybody could find) were not responding to DNS queries. It appears that Akamai started to come back online at around 10:00 AM EDT.

    Since a great many big name sites use Akamai, this effectively made large parts of the Internet unreachable. The destination servers themselves were up, but clients were unable to turn names (like www.example.com) into network addresses (like 192.0.2.42).

    As Akamai maintains dozens, if not hundreds, of DNS servers across the globe, it is extremely unlikely that this was due to a normal equipment failure or DoS attack. Some kind of internal system trouble is much more likely. Whether a deliberate attack, or an accident, is unknown to me at this time. It could just be an internal configuration change blew up in a really bad way. Sh*t happens.

    I do not know if this was just an Akamai DNS problem, or if other Akamai services were also affected.

    Due to the way Akamai is usually implemented, it happened that, in many cases, the second-level domain names (like example.com) worked, but subdomains (like www.example.com and mail.example.com) did not. This is because most organizations put in CNAME records (pointing to names in *.akadns.net) for the subdomains. You cannot use a CNAME record for a domain that has other records, though, so most domains still had traditional A records, on their own nameservers, at the second-level.

    The following sites/organizations are known to use Akamai: Yahoo, Google, Microsoft, Altavista, FedEx, Xerox, Apple

  • Reminds me of a story (Score:5, Interesting)

    by Venner ( 59051 ) on Tuesday June 15, 2004 @10:24AM (#9430166)
    Not too long after 9/11, I was surfing the net and needed to look up something at the Library of Congress for one of my classes. It wouldn't connect. At first I thought we'd just lost DNS (not so uncommon an occurance at my university in those days), but found I could still connect to slashdot.org and some other sites.

    Being a geek, I thought up a list of about 30 sites to ping, scattered across the US. (.govs and .edus mostly.) The ones that replied, I plotted on a US map based on their DNS LOC. (A project I wrote for a previous class.)

    I freaked out a bit when the mid-atlantic seaboard came up missing. I crossed my fingers hoping that it was just some idiot who'd accidently cut one of the main fibers (which it what it ended up being) and not that Washington DC was now a big hole in the ground.

  • From Akami's Page (Score:4, Informative)

    by esconsult1 ( 203878 ) on Tuesday June 15, 2004 @10:33AM (#9430283) Homepage Journal
    Some info from Akami...

    # Maximizes e-business revenue by guaranteeing 100% availability

    EdgeSuite Enterprise Edition is built on the globally distributed and highly scalable Akamai EdgePlatform, comprising over 14,000 servers deployed in over 1,000 networks across more than 70 countries. With this global reach, users can deliver their content from the edges of the Internet - closest to their users.

  • How Sites are Coming Back Online (Score:5, Informative)

    by TheSync ( 5291 ) on Tuesday June 15, 2004 @10:35AM (#9430306) Journal
    From NANOG [merit.edu] mailing list again:

    Google pulled references for akamais dns servers a short period ago. they are presently serving their own dns requests.

    Also:

    People seem to be getting around this by changing their DNS entries.

    E.g. www.yahoo.com always used to be a CNAME for www.yahoo.akadns.net. But
    now:

    # host www.yahoo.com
    www.yahoo.com is an alias for www.dcn.yahoo.com.
    www.dcn.yahoo.com has address 216.109.118.64
    www.dcn.yahoo.com has address 216.109.118.65
    www.dcn.yahoo.com has address 216.109.118.66
    www.dcn.yahoo.com has address 216.109.118.67
    www.dcn.yahoo.com has address 216.109.118.68
    www.dcn.yahoo.com has address 216.109.118.69
    www.dcn.yahoo.com has address 216.109.118.70
    www.dcn.yahoo.com has address 216.109.118.71
    www.dcn.yahoo.com has address 216.109.118.72
    www.dcn.yahoo.com has address 216.109.118.73
    www.dcn.yahoo.com has address 216.109.118.74
    www.dcn.yahoo.com has address 216.109.118.75

    Which is owned by Yahoo! (via HotJobs.com).

  • Whatever happened to my decentralized net... (Score:4, Insightful)

    by Lord Kano ( 13027 ) on Tuesday June 15, 2004 @10:35AM (#9430309) Homepage Journal
    Whatever happened to my decentralized net with no single point of failure?

    Outsourcing and consolidation.

    LK

  • Hmm . . . (Score:3, Interesting)

    by npsimons ( 32752 ) on Tuesday June 15, 2004 @10:40AM (#9430378) Homepage Journal

    "Checking all my favorite sites this morning, I saw that about half a dozen seem to be offline. Trying to figure out why, I found an interesting article on the front page at http://isc.incidents.org/. Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others. Whatever happened to my decentralized net with no single point of failure?"

    Hmmm, corporate whore much? Slashdot, Debian and my own two sites seem to be working just fine. Maybe the sites you choose to visit just don't get the 'net and it's decentralized nature.

  • it's like your rights, you can sign them away (Score:3, Interesting)

    by swschrad ( 312009 ) on Tuesday June 15, 2004 @10:48AM (#9430463) Homepage Journal
    and folks often do... witness the onerous "personal contracts" you have to sign to get into the music business, where you are essentiall a creative wage slave and don't own your stuff. non-compete and discoveries-belong clauses in your work contract also sign your rights away to The Man. similarly, if you register your DNS information independently and run your own servers, your ISP and its uplines do the same, and so on including all the sites you visit, you theoretically should not be captive to any of the commercial DNS services.

    as I understand it, akamai is a distributed content hosting/caching service that also does DNS server services. they put a blade in your local ISP under contract, and popular pages from their customers serve off the local akamai server cache. they handle the DNS for those sites as I understand. if their blade caches get fed evil data, you get evil data, and www.fartblossom.org may disappear.

    you can kill DNS by screwing up your own router, too. lots of ways to kill a distributed service that requires everybody to cooperate on a common set of standards and parameters.

  • Created SPoF (Score:5, Interesting)

    by Todd Knarr ( 15451 ) on Tuesday June 15, 2004 @10:58AM (#9430615) Homepage

    The problem is that those sites created their own single point of failure by all using Akamai for DNS. When Akamai DNS fails, sites that depend on it for their own DNS fail.

    It used to be nearly impossible for this to happen. The original rules for DNS were that you had to have at least 2 nameservers for your domain, preferrably 3 or more, and they couldn't be on the same physical networks. With that rule having a single network go down rarely made any domain unresolvable (backbone networks whose outages could render dozens or hundreds of other networks unreachable being the exception). Maybe we should put the old nameserver-diversity rules back into place.

  • "Caught in a BIND" (Score:4, Informative)

    by stock ( 129999 ) <stock@stokkie.net> on Tuesday June 15, 2004 @11:20AM (#9430936) Homepage
    Jon Lasser predicted some troubles long time ago : http://crashrecovery.org/bind9.html [crashrecovery.org] . His article is on http://theregister.co.uk/content/55/28235.html [theregister.co.uk] and titled "Caught in a BIND".

    Robert

Slashdot Top Deals

Technology is dominated by those who manage what they do not understand.

Close