Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Communications The Internet

Tier One ISPs Dying 394

Posted by CowboyNeal from the they're-just-faking dept.
xbmodder writes "Two tier one ISPs are down today. At about 23:30PST both Verio and Level 3 starting having problems with routes. According to Level 3 this is a software upgrade gone awry. Is this the end for Level 3?" Many, many reports about this are coming in, and if you're wondering why the stories were rather sparse overnight, it's because it's difficult to post them without internet access. Hope everyone else is back online too.
This discussion has been archived. No new comments can be posted.

Tier One ISPs Dying More

Tier One ISPs Dying

Comments Filter:

  • Could be good (Score:5, Funny)

    by sdo1 ( 213835 ) on Friday October 21, 2005 @07:48AM (#13843355) Journal
    Maybe I'll get some work done today for a change.

    -S

  • Flicker (Score:5, Funny)

    by Oculus Habent ( 562837 ) * <oculus.habent@gma i l . c om> on Friday October 21, 2005 @07:48AM (#13843357) Journal
    Is there a term for this kind of intermittant site inaccessability due to Internet outage -- not the user or the server being offline, but the Internet failing?
    • You mean something like "connection loss"?

    • Re:Flicker (Score:2, Funny)

      by Cerberus7 ( 66071 )
      Yeah, it's called an "Oops!" Now somebody needs to make an acronym for this type of event using those letters. The two "O"s next to each other are giving me a hard time.

    • TEOTWAWKI (Score:5, Funny)

      by D4C5CE ( 578304 ) on Friday October 21, 2005 @08:01AM (#13843424)
      Is there a term for this kind of intermittant site inaccessability due to Internet outage -- not the user or the server being offline, but the Internet failing?
      Yes. Domesday as predicted in the ancient scrolls. In this day and age it is commonly called The End of the World as We Know It.
    • Luckily, seems no. At least this happens way more rarely than a Broken Arrow to get a name.

    • flapping (Score:5, Interesting)

      by SpectralDesign ( 921309 ) on Friday October 21, 2005 @09:00AM (#13843689)
      Way back in the day when I was a Network Controller at BBN Planet, if we began to have cascading routing outages we'd call it "Flapping"... Visualize a wounded bird squirming around on the ground flapping...

      Takes me back... My first night on the job a rat in Berkeley chewed through the wrong cable and got himself fried -- he also happened to take the entire west-coast off the internet for the better part of a day.

      Then there was the time an electrical worker got vaporized in a hole near MIT which caused quite a problem too as it overloaded the MIT power station, but the fallout wasn't nearly as bad as the day of the rat...

      • Clarification (Score:5, Informative)

        by Just Some Guy ( 3352 ) <kirk+slashdot@strauser.com> on Friday October 21, 2005 @10:29AM (#13844249) Homepage Journal
        I'm sure you know this, but for the rest: "flapping" is the common term for when a router's routing tables rapidly cycle between two invalid states [cisco.com]. The dead bird analogy is pretty descriptive, but the term "flapping" has technical and not allegorical origins.

    • Re:Flicker (Score:3, Funny)

      by StikyPad ( 445176 )
      I dunno, but for the first time ever, thousands of users were correct when they called tech support to claim that "the internet is broken."

  • Outtage Explained (Score:5, Funny)

    by MMyers5 ( 589677 ) on Friday October 21, 2005 @07:48AM (#13843358)
    It's nice to see something explaining why I was paged at 2:30am. And now, to whom from Level3 do I send my bill?

  • Call me silly? (Score:4, Funny)

    by Killjoy_NL ( 719667 ) <slashdot@@@remco...palli...nl> on Friday October 21, 2005 @07:48AM (#13843359)
    But what is a tier 1 ISP?

    Is that like a bandwidth wholesaler or something?

    • Re:Call me silly? (Score:4, Informative)

      by rylin ( 688457 ) on Friday October 21, 2005 @07:51AM (#13843374)
      One that doesn't lease their infrastructure.
      Eg. you have your own large backbone, you own all your equipment.

      In effect, a small and wholly owned internet that peers with other internets.
      • Thanks :)

        (hehe he said internets, George is that you?) ;)

      • Re:Call me silly? (Score:3, Informative)

        by jusdisgi ( 617863 )
        That's not accurate. Lots of tier 2 and lower providers own their infrastructure. The important qualification of being a tier 1 ISP is that they don't pay anyone else to exchange traffic with them. The tier 1 guys are all predicated on the idea that they are huge enough that none of the others of them can afford to not have good and direct peering with them. Level3 can't afford to not be peered with MCI, and MCI can't afford to not be peered with AT&T, etc. So they all peer for free with each other. The

      • Re:Call me silly? (Score:3, Informative)

        by porky_pig_jr ( 129948 )
        This is not correct. Tier 1 ISP has nothing to do with leasing or own telco. I have worked for Tier 1 ISP which did not own any of its telco lines. Everything was leased from different companies: MCI, AT&T, GTE. (hint: that ISP had AS 1).

        The way Tier 1 ISP is defined is mostly by its magnitude. At the time I've worked for that ISP, the rough rule of tumb was that Tier 1 ISP must have a few large capacity pipes from coast to coast at least. Must carry enough traffic so other Tier 1 ISPs can exchange the

    • Re:Call me silly? (Score:5, Informative)

      by CvD ( 94050 ) * on Friday October 21, 2005 @07:52AM (#13843378) Homepage Journal
      Tier 1 are the huge ISPs, which peer with eachother (and don't pay eachother transit fees) and sell transit services to smaller ISPs (which do pay fees to send traffic through the Tier 1 ISPs). So yeah, bandwidth wholesalers is pretty accurate. See this wikipedia article [wikipedia.org].

    • Re:Call me silly? (Score:5, Interesting)

      by Anonymous Coward on Friday October 21, 2005 @08:50AM (#13843638)
      In 1994, The National Science Foundation (NSF) awarded contracts to replace the National Science Foundation Net (NSFNet) Internet backbone. These contracts were for backbone transport, routing arbiter and traffic exchange points (NAPs).
                These contracts were awarded for the original 15 NSF sponsored NAPs, and to become a Tier 1 ISP, you had to have atleast DS3 connectivty to all 15 NAPs.
                It's a very old and crappy definition, and I wish people would stop using it, because it is very easy to meet now adays, and most of those original NAPs are now insignificant, compared to the power of the force.

      • Re:Call me silly? (Score:3, Interesting)

        by w4pso ( 874168 )
        My understanding is that a Tier 1 ISP is now defined as one that has established free peering points with all other Tier 1 ISPs.

        Considering that free peering is likely only established between 2 networks that have close to a 1:1 bit exchange, this is a very high bar to meet.

  • What is this about? (Score:3, Insightful)

    by PlatinumX ( 924555 ) on Friday October 21, 2005 @07:49AM (#13843360) Homepage
    An ISP's server being down 1 day is unacceptable of course, but to say it is dying already? or is there more to these ISP's? (haven't heard of them before)

    • Re:What is this about? (Score:5, Informative)

      by neosake ( 655724 ) on Friday October 21, 2005 @07:56AM (#13843397) Homepage
      This is not _just_ their server being down, this is the entire network of two tier 1 carriers [wikipedia.org].

      The (basic) implications of this is that a good chunk of the internet as a whole is inaccessible to the rest of the internet.

    • Re:What is this about? (Score:5, Informative)

      by raddan ( 519638 ) on Friday October 21, 2005 @08:18AM (#13843494)
      We're not talking about just a server. We're talking about the entire ISP's networking capability. Tier 1 ISPs own huge swaths of networks-- literally miles and miles of cable, and sometimes radio and other links. They route the traffic across these lines.

      When a Tier 1 provider goes down, their customers go down too. That picture on the Boing Boing page shows a list of the Tier 1 providers. Every ISP that is NOT a Tier 1, gets their access from a Tier 1.

      People speculate that Level 3 is dying because they've been making some really bad decisions lately, resulting in a lot of outages. A couple of weeks ago, they actively filtered out traffic from their competetor, Cogent, over a dispute from how much to charge at the point their networks exchanged traffic (called a 'peering point'). Now this. The rumor is that the company is in financial trouble.

      • Re:What is this about? (Score:4, Informative)

        by Alioth ( 221270 ) <no@spam> on Friday October 21, 2005 @08:22AM (#13843511) Journal
        It's not a rumour that Level 3 is in financial trouble - it's clear for all to see. They have crushing debt repayments right now.

        The Cogent spat isn't over yet either - Level 3 are going to de-peer Cogent again on November 9th. They are trying to force Cogent to pay for transit, but right now it looks like Cogent holds the strongest hand and Level 3 will be once again forced to back down.

        • Re:What is this about? (Score:5, Informative)

          by HavocBMX ( 760265 ) on Friday October 21, 2005 @11:29AM (#13844763)
          The reason that Level 3 isn't happy with the peering arrangement currently is that it's not even remotely even. Level 3 sends almost nothing over Cogent's network and Cogent sends over a vast majority of their traffic through Level 3. A peering agreement is based on the premise that the companies will be sending almost equal amounts of traffic through each network. Level 3 has been analyzing that for a time now but the last straw was when Cogent had a sales blitz targeting Level 3 customers saying that they would dramatically drop their prices to almost nothing to get them to switch away from Level 3. They are now also using the downtime that was experienced due to the peering problem in their advantage even though Cogent is in the wrong. Cogent knew about the depeering and did nothing to resolve it.

          • Re:What is this about? (Score:4, Informative)

            by Alioth ( 221270 ) <no@spam> on Friday October 21, 2005 @11:39AM (#13844856) Journal
            Oh yes, I'm aware of all of that - but (generally speaking) Cogent has the content, and Level 3 has the users. Guess who catches the most heat from the de-peering from its customers - Level 3 - as their customers will tend to see the problem first.

            I predict that Cogent will do the same again as well - not lift a finger to fix the problem when they are de-peered on November 9th, and Level 3 will probably end up being forced to re-peer as customers whine that they are not getting the whole Internet and threaten to take up Cogent's free 1 year offer.

      • Re:What is this about? (Score:5, Informative)

        by Jugalator ( 259273 ) on Friday October 21, 2005 @09:17AM (#13843775) Journal
        Tier 1 ISPs own huge swaths of networks-- literally miles and miles of cable, and sometimes radio and other links. They route the traffic across these lines.

        More precisely, Level3 seem to own 23,000 miles of optic fiber [prnewswire.com]. :-)

        The rumor is that the company is in financial trouble.

        Yeah, not so much of a rumor anymore either -- Level 3 loss widens [reuters.com].
      • Every ISP that is NOT a Tier 1, gets their access from a Tier 1.

        true in a sense but highly misleading. A large tier 2 isp is likely to have uplinks to MULTIPLE tier 1 providers as well as many peering links of thier own.

  • Guess not (Score:5, Informative)

    by springbox ( 853816 ) on Friday October 21, 2005 @07:49AM (#13843362)
    Take a look at the scoreboard [keynote.com] now. The mentioned problems are gone and Level 3 is no longer in the red.

  • huh?? (Score:2, Insightful)

    by Anonymous Coward
    Why would a software upgrae going wrong be the end of a gigantic Tier-1 ISP?
  • Today i was playing world of warcraft and on our raid about 25% of my guild mates lost their internet on and off. Other than that the lag was higher than normal but i wondered what the hell was going on. Anyway we still pwn some dragons in BWL :)

  • Noticed it this morning (Score:5, Interesting)

    by discord5 ( 798235 ) on Friday October 21, 2005 @07:56AM (#13843400)
    Noticed this this morning when a customer called upset about his hosting services being unreachable. A quick traceroute showed one of level3's ip to be down. A few minutes later more customers had problems with different routers from level3. As soon as I saw level3 I knew enough, shrugged it off and told the customer that it was routing problem we couldn't fix but those responsible were most likely already trying to fix it.

    It seems fixed now though, so no, this isn't the death of the Internet just yet.
  • Has Netcraft confirmed this?

  • This just in... (Score:4, Funny)

    by mtec ( 572168 ) on Friday October 21, 2005 @07:58AM (#13843413)
    The reports of my death have been greatly exaggerated.
    Tier One

  • /.'ed (Score:5, Funny)

    by connah0047 ( 850585 ) on Friday October 21, 2005 @08:02AM (#13843429)
    Hey look, we slashdotted Level 3!

  • I noticed it. (Score:3, Informative)

    by HoneyBunchesOfGoats ( 619017 ) on Friday October 21, 2005 @08:04AM (#13843434)
    I was up late studying for a German exam, and I was having problems connecting to websites hosted in Germany that I was using to help myself review (dict.leo.org and canoo.net, if you're curious). US websites worked no problem.

    Off to the test!
    • This is like the 21st century equivalent to pulling the fire alarm in school in order to get out of taking a test.

      I can't take the test, professor. The sites with the study guides were Slashdotted this morning!

      You deserve an A for shrewdness.

  • Non event... for now (Score:5, Informative)

    by elfguygmail.com ( 910009 ) on Friday October 21, 2005 @08:12AM (#13843467) Homepage
    While this only lasted a few hours, it still caused a mess across the North American Internet during those hours. The point is a small amount of big networks are responsible for over 90% of the traffic on the Internet. If alter.net went down it would be total chaos. If just one of the major peering points went down, sure the traffic would be rerouted, but overloading the other points at such high latency that it would be almost unusuable. You better hope no one destroys MAE-EAST or we'll have a live example of what ife without the Internet is like.

  • Link in the article (Score:4, Funny)

    by Alphabet Pal ( 895900 ) on Friday October 21, 2005 @08:16AM (#13843484)

    I notice the article links back to Slashdot... I wonder is Slashdot is going to get BoingBoing'ed?

  • What kind of Timeframe (Score:4, Interesting)

    by PhraudulentOne ( 217867 ) on Friday October 21, 2005 @08:21AM (#13843505) Homepage Journal
    Is this even an issue? I mean, this was probably scheduled maitenance that went a little longer than expected. I have been through this before. It just sounds like Level 3 dropped some core routers for a few minutes to do a code upgrade - it didn't work so hot, so they were down for a few more mintes, OSPF/BGP decided to tell all the clients that they have no routes, Level 3 gets the routers back up, OSPF/BGP tells everyone that their fine again. Was this like 6 hours, or 45 min?

    • Re:What kind of Timeframe (Score:4, Informative)

      by Jugalator ( 259273 ) on Friday October 21, 2005 @09:11AM (#13843748) Journal
      It was maybe 2 hours or so before new routing tables started spreading to bypass Level3's and Verio's networks, and afterwards it started stabilizing again, then it seems Level3 has since then woke up again. The XO network also had routing troubles from this btw, maybe more too. Sites and services such as AOL, SpeakEasy (when asked, they were stumped and could only say it affected all their customers, hehe), Google, and Wikipedia had access problems depending on where you lived during this timeframe.

    • Re:What kind of Timeframe (Score:3, Informative)

      by Anonymous Coward
      I dropped my BGP session to Level3 but they did not retract the routes, so not only could they not route my packets but they claimed (via the routing table) that they still could. From my vantage point (Chicago) the problem was resolved in about an hour,
  • Is this the first time this has happened? Is it too early to start talking about re-thinking the way this is put together?

  • So much for the internet surviving a nuclear war! (Score:4, Insightful)

    by Viol8 ( 599362 ) on Friday October 21, 2005 @08:27AM (#13843529) Homepage
    Now it can't even survive a software upgrade on some of the routers!

  • Is the Internet Down? (Score:5, Funny)

    by PortWineBoy ( 587071 ) on Friday October 21, 2005 @08:28AM (#13843536)
    Why couldn't this have happened during my business day? For just once when a user calls and asks "is the internet down?" I'd like to be able to say "actually, yes, it is."

  • Overlay Routing (Score:5, Informative)

    by omnirealm ( 244599 ) on Friday October 21, 2005 @08:53AM (#13843651) Homepage
    This sort of event provides motivation for overlay routing schemes, which can compensate for major outages along various routes of the backbone:

    http://www.usenix.org/events/nsdi04/tech/full_pape rs/subramanianOver/subramanianOver.pdf [usenix.org]
    http://www.eecs.umich.edu/~farnam/pubs/2005-hwj-in focom.pdf [umich.edu]

  • Time in UTC (Score:3, Informative)

    by GerritHoll ( 70088 ) <gerrit@nl.linux.org> on Friday October 21, 2005 @08:53AM (#13843654) Homepage
    23:30 PST = 07:30 UTC = 09:30 CEST

    But perhaps what's really meant is:

    23:30 PDT = 06:30 UTC = 08:30 CEST ?

  • Over 100 comments so far.... (Score:3, Funny)

    by mblase ( 200735 ) on Friday October 21, 2005 @08:59AM (#13843684)
    ...and not one Netcraft joke?

  • The main problem areas... (Score:4, Interesting)

    by dantheman82 ( 765429 ) on Friday October 21, 2005 @09:18AM (#13843776) Homepage
    happened in Detroit in the last 24 hours. Apparently all ingoing/outgoing traffic to other Tier One ISPs had problems in that city. Also, Philadelphia had really slow traffic within Level3 (and slower to all the others), and had major problems connecting to Verio. San Diego also had some problems, especially within the Level3 network. St. Louis was the only area without major problems...

    For a breakdown, check out this view of the data [keynote.com].

  • Microsoft? (Score:3, Funny)

    by Frankie70 ( 803801 ) on Friday October 21, 2005 @09:23AM (#13843808)
    Is there any way we can blame Microsoft for this?
    Were they upgrading to one of the Beta builds of Windows Vista Home Edition?

  • You insensitive clod! (Score:3, Funny)

    by Anonymous Coward on Friday October 21, 2005 @09:30AM (#13843853)
    I'm not back online

  • Level 3? (Score:3, Interesting)

    by Greyfox ( 87712 ) on Friday October 21, 2005 @09:52AM (#13843975) Homepage Journal
    Haven't they been hanging on by their fingernails since the dot-com bust? I've know a few guys who got burned working for them just before the bust, and I've seen several recruiters post stuff like "A local communications company (NOT Level 3!)" in their job reqs.

    I don't know that they've replaced Sprint yet on my list of most sucktastic internet companies. Time was you lost connectivity to an important piece of the Internet (Like your favorite Quake TeamFortress server) and a traceroute would show the failure somewhere in the Sprint backbone. So far they've been more reliable than Sprint at their worst, at least for me.

    If they go under, well Tier 1's don't ever really die. Chances are one of the other Tier 1's will buy their assets and it'll be business as usual. Usually the buyer is MCI.

    Of course the true test is pretty easy -- has anyone who works at Level 3 had their paycheck bounce yet? Surely there are a few readers among their employees...

    • Re:Level 3? (Score:3, Insightful)

      by RPoet ( 20693 )
      If they go under, well Tier 1's don't ever really die. Chances are one of the other Tier 1's will buy their assets and it'll be business as usual. Usually the buyer is MCI.

      And with fewer and fewer tier 1's, is it any wonder things like this happen?

  • FFS, what a fucking dreadful summary (Score:5, Insightful)

    by Cally ( 10873 ) on Friday October 21, 2005 @10:00AM (#13844039) Homepage
    Is this the end for Level 3?

    No, of course not, you blithering imbecile. L3 had a 2 hour global routing meltdown. Now, it's fixed. Whilst their routes were flapping, other carriers saw transient increases in latency and some problems with reachability, to some sites. However, everything continued to work properly for non-L3 customers. Two hours later L3's routes are back and working properly. End of story, nothing to see here, move along please.

    Slashdot editors, do you really expect us to believe that no-one had submitted a more coherent or accurate story than this one? Come on, for heaven's sake.

    Anyway, a network engineer's view can be seen in the overnight traffic on NANOG: http://www.merit.edu/mail.archives/nanog/2005-10/ [merit.edu] "Tier One ISPs dying" indeed. Worst. Story. EVER.

    • "No, of course not, you blithering imbecile. L3 had a 2 hour global routing meltdown. Now, it's fixed."

      However, L3 has been having "issues" this month that have left a lot of lower-tier ISPs in the uncomfortable position of explaining to their customers "We know the internet is down but there's nothing we can do about it." This outage really can't be good for their reputation, and I can see more potential customers taking their money elsewhere because of this.

      Just because the technical issues have been fix

      • However, L3 has been having "issues" this month that have left a lot of lower-tier ISPs in the uncomfortable position of explaining to their customers "We know the internet is down but there's nothing we can do about it." This outage really can't be good for their reputation, and I can see more potential customers taking their money elsewhere because of this.

        Just because the technical issues have been fixed doesn't mean their finances have been fixed as well.

        See now that's a story, if you add a couple

    • "Tier One ISPs dying" indeed. Worst. Story. EVER.

      Come on, this is SlashDot.

      It should be:

      "Tier One ISPs dying" indeed. \/\/0rst. 570ry. EVAR.

  • X is Dying (Score:5, Funny)

    by Christianfreak ( 100697 ) on Friday October 21, 2005 @10:06AM (#13844075) Homepage Journal
    Glad to see that Tier 1 ISPs are joing the ranks of BSD and Apple.

  • Has to be said... (Score:3, Interesting)

    by KC7GR ( 473279 ) on Friday October 21, 2005 @11:35AM (#13844820) Homepage Journal
    Oh, so THAT's why my daily spam load suddenly dropped by about 35% or so...

  • Unixshell graphs caught my eye (Score:3, Informative)

    by tezza ( 539307 ) on Friday October 21, 2005 @12:21PM (#13845257)
    I don't work there, or own any shares in unixshell.

    I was looking for a Linux Virtual Host, blah, blah.

    Stumbed apon these pretty pictures [unixshell.com] (near bottom of page) .

    Curious, I thought, what happened to Level(3) ? I though for a second because perhaps unixshell had a peering with those people that Level(3) were in dispute with.

    Nope, just one of those regular outages that make the 99.999% promises sound a little over done.

Slashdot Top Deals

The optimum committee has no members. -- Norman Augustine

Close