Forgot your password?
typodupeerror
The Internet

Network Blackout 183

Posted by michael
from the no-see-um dept.
An anonymous reader writes "Renesys put together a special report on the effects of the recent blackout on routing and network reachability on the Internet. It includes a cool animation of networks dropping off the internet (presumably as a result of the power outage). It is interesting to see how localized some of the outage was--networks in New York state right up to the Vermont border go dark while everything on the other side of the border is quiet. New York City obviously gets clobbered."
This discussion has been archived. No new comments can be posted.

Network Blackout

Comments Filter:
  • by Purosesuchi-Zu (695676) on Tuesday August 19, 2003 @06:57PM (#6738907) Journal
    That little red dot at the tip of LI is my home LAN going down...
  • Mirror (Score:2, Informative)

    In case the site is slow, for whatever reason, here are a couple mirrors for link 2 [martin-studio.com] and link 3 [martin-studio.com].
  • by Anonymous Coward on Tuesday August 19, 2003 @07:00PM (#6738943)
    The blaming and finger pointing began almost as quickly as the lights went out. First it was the U.S. and Canada blaming each other for causing this particular blackout, but inevitably the blame conversation turned to larger issues of policy, and how something like this could happen in such a heavily regulated industry.

    Some of the finger pointing in the national press has been at deregulation -- if it weren't for deregulation, we would be better able to control and manage the grid. This misguided contention is incorrect in a number of ways.

    First, the "deregulation" that has occurred in electricity has primarily been in opening up wholesale markets for power generators and their customers (i.e., utilities), enabling people in Manhattan to continue consuming power (and clamoring now for more regulation) without Con Edison having to build more power plants on the island itself. The existence and growing vitality of wholesale electricity markets has created substantial value in the past decade, through encouraging generation where it is cheapest and sales of power to where it is most needed.

    But this limited amount of market liberalization has left the industry in an awkward place. Generation is largely governed by market processes, but transmission and retail distribution remain heavily regulated. The investment decisions of transmission owners and the retail rates that they can charge to their end customers all hinge on rate cases that are decided by state-level regulators. The rates that regulators allow take into account changes in costs, required investments, and the payment to the utility of a rate of return on the assets they own. For much of the past decade this rate of return has been substantially lower than what utilities could earn from doing other things with their money, so they did not invest in building much new transmission capacity or in upgrading existing lines. Nor did a regulatory environment that is a relic from the 1930s, constructed to govern and control local, vertically integrated utilities, either have the incentive or the wherewithal to force the utilities to invest in transmission assets that would carry power to customers in other states.

    This lack of investment in the infrastructure that carries the product exchanged in growing, vibrant wholesale electricity markets has become a problem -- not an overnight problem, as those who follow the industry have been concerned about transmission capacity for at least five years. The numbers offered this weekend suggest that electricity volume has increased 30 percent while transmission carrying capacity has increased only 15 percent. This fact illustrates the mismatch between the dynamic markets for wholesale power and the rigid, maladaptive set of state-level regulations and incentives that govern transmission investment decisions.

    Markets adapt to changing conditions. The existing electricity regulations do not, and because of that, the transmission infrastructure has not adapted to the increased demand on it from the increasing vibrancy of wholesale electricity markets.

    So how do we proceed to ensure that a blackout of this magnitude does not happen again? There are four things that can relieve the strain on the grid. The knee-jerk reaction of many people is "build more wires!" More wires will increase the carrying capacity of the system, and in some cases transmission owners can add lines to existing paths. But this approach faces some serious obstacles -- such construction is expensive and time-consuming. Most importantly, though, getting new lines and towers sited is increasingly difficult, as people and communities object to having such large structures near them or strung overhead.

    A second option is to use new technologies, such as high-temperature superconductors and sophisticated computer switching, to upgrade the capacity of the existing power lines. While also expensive, this option gets around the NIMBY issues that accompany the siting of new lines.

    A third option is to build
    • Security focus is saying [securityfocus.com] that a previous outage wasn't deregulation, wasn't transmission, it was the slammer worm hitting Ohio's Davis-Besse nuclear power plant and disabling a safety monitoring program.

      Interesting story. CNN and a bunch of people on NANOG (www.nanog.org) were speculating that this outage was caused by the msblaster worm. This story backs up at least the feasibility of that.

  • by binaryDigit (557647) on Tuesday August 19, 2003 @07:04PM (#6738973)
    Everyone could have switched over to ConMars power. Being a Russian installation, they would have just strung about 17million Home Depot Heavy Duty extension cords all plugged together from here to Mars (the outdoor version of course) to access it.
  • by Exiler (589908) on Tuesday August 19, 2003 @07:04PM (#6738977)
    Is that a red circle on the webserver hosting that gif?
  • Apology (Score:5, Funny)

    by ticklemeozmo (595926) <(justin.j.novack) (at) (acm.org)> on Tuesday August 19, 2003 @07:05PM (#6738981) Homepage Journal
    I'm sorry. I had apparently text'd most of my friends saying electricity was poor, and almost immediately word spread around the north east..
  • backup? (Score:5, Insightful)

    by killermal (545771) on Tuesday August 19, 2003 @07:05PM (#6738987)
    Don't these backbone routers have backup? I was in an ISP server house in the UK which had a full backup system. In the case of a power failure, it had a UPS that kicked in for 10 seconds while the generator was booting up, which then provided power for the infrastructure of the building. I would find it hard to believe that in the USA they don't have similar systems?
    • Re:backup? (Score:5, Funny)

      by soliaus (626912) on Tuesday August 19, 2003 @07:08PM (#6739016) Homepage Journal
      I would find it hard to believe that in the USA they don't have similar systems?

      We prefer the rat on a wheel approach to power backup. It usually works, we just forgot to feed them so it failed on such a wide spread application

    • Re:backup? (Score:1, Informative)

      by Anonymous Coward
      Well...I don't think this is limited to backbone routers, and there are a LOT of routers out there...
    • by Baron_Yam (643147) on Tuesday August 19, 2003 @07:12PM (#6739056)

      Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.

      Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.

      • by notque (636838) on Tuesday August 19, 2003 @07:16PM (#6739085) Homepage Journal
        I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.

        Seriously, you've got to get out more.
        • Well, to be fair, government officials were all over the radio telling us NOT to go out at the time, since there were no traffic lights, street lights, gas stations, etc up and running.

          Honestly, traceroutes are not my everyday source of entertainment!

      • Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.

        Why would you want it all in batteries? Use the UPS to tide you over until you can fire up the gas/diesel generator. Those you can get pretty cheaply (well, compared to 300lbs of batteries) and are useful for other thin
      • by BigBlockMopar (191202) on Tuesday August 19, 2003 @08:18PM (#6739521) Homepage

        Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.

        Yeah, same up here in Ottawa, Canada... I was awakened early on Friday morning to the sound of my servers POSTing; my power was back in under 12 hours. I was lucky. :) (Made sure to double-check that hdparm was set to spin down the drives, that and killing the A/C were my contributions to energy efficiency.)

        Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.

        I don't have a UPS (well, I do, I got one free, but it's broken and I haven't had time to troubleshoot it - anyone got schematics for an APC Back-UPS Pro 280?), so your mileage may vary. If the UPS runs off 12V batteries, you might be able to:

        • cobble a set of binding posts onto the side of its case, in parallel with the battery, and connect them to the battery in a running car. (Essentially, "jump-start" your UPS. Start the car first.)
        • Replace the 12V gel-cel battery with a good old-fashioned car battery. Even a weak used one should run it for a lot longer, but I haven't seen how charge current is regulated when the UPS is on AC, so I don't know how well the UPS's charging circuits will tolerate it.
        • Scoop an old gas lawnmower out of the garbage (Briggs and Stratton or Techumseh 4-stroke motors are preferable and very reliable if you keep them well tuned). Fix it, and install a pulley where the blades were (usually a 3/4" or 1" keyed shaft, and you want to take a 4L belt of the required length). Cut a hole in the deck, install a bracket, and hang a GM 1-wire alternator (1975-1985 models) in there. The lawnmower's deck is ground, the plastic-insulated bolt on the back is the positive. Mount a car battery onto it, and you have a portable jump-starter and 12V generator. Good also to weld on a perch for your toolbox. (Built one for myself, works *great* in junkyards when you want to test compression or oil pressure in an old car.)

        Note that I don't know how the UPS's inverter will handle running at rated load for longer than the internal battery is capable, nor do I expect that the UPS will have much noise suppression on the battery leads - after all, batteries themselves are pretty much noise-free electrical sources and alternators are not.

        • You wouldn't be that guy from the redgreen show by any chance ?
          • by BigBlockMopar (191202) on Tuesday August 19, 2003 @09:10PM (#6739858) Homepage

            You wouldn't be that guy from the redgreen show by any chance ?

            Steve Smith? No. But we are on first name basis (no kidding!) and I wear lots of flannel. He once did something on the show which really reminded me of snowblower couch races with friends, and I can't remember if I told him about it or not...

            (Snowblower couch racing? You scoop sofas, mattresses and box-springs from the garbage and store them in your backyard. You buy beer and sharpen the ice-cutters on your snowblower. Then you invite over some friends with their own snowblowers and see who can demolish the upholstered furniture the fastest. Truly a good reason to own a snowblower in a warm climate.)

          • I see no mention of duct tape in the grandparent's post, so it's pretty safe to say that it's not him.
        • Note that I don't know how the UPS's inverter will handle running at rated load for longer than the internal battery is capable

          I picked up a Sola 300 UPS for $10 with 2 dead batteries, so bought 2 12v car batteries. I killed the first UPS after about an hour of testing (at the rated 300 W load), because the main transformer got too hot and burnt out one of the windings. The transformer retained a heckload of heat for several hours.

          So I bought another $10 UPS, and this time drilled holes all through the c
        • I like the lawnmower idea, and would only add one thing to it - if you run the 12V output of the alternator through an inverter to create AC, then plug the UPS into the inverter you get line conditioning at the inverter inputs (because they're usually designed to run in a car, which has an alternator, which has the noisy DC), plus you don't have to hack open a perfectly good UPS.

          You do lose a bit of power converting AC-DC-AC-DC-AC (Mower Alternator - Car Inverter - UPS Inverter - UPS Battery - UPS Inverter

          • I like the lawnmower idea, and would only add one thing to it - if you run the 12V output of the alternator through an inverter to create AC, then plug the UPS into the inverter you get line conditioning at the inverter inputs (because they're usually designed to run in a car, which has an alternator, which has the noisy DC), plus you don't have to hack open a perfectly good UPS.

            Except that UPSs (at least some of them) don't like the power produced by inverters. I have a customer with an alternative powe
      • my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours

        I know we are talking about places like New York, but does nobody have a car??? That beauty is probably 100x the power of your UPS battery, and it comes connected to a mobile, gasoline-powered 12V generator, on wheels.

        If you don't like doing some wiring yourself (to the UPS), 300watt power inverters are under $50, which is more than enough to keep all your lights on, and a TV, radio, possibly computer+monitor, etc

        • Whereas a 200HP car engine can put out nearly 150KW of power at peak, which is enough to run several suburban homes, the alternator can probably only put out around 1KW max. Be careful not to overload and burn it out, costing you more money than a couple extra batteries would have. }:)

          Cars are designed to haul themselves and your ass around, not keep your little server farm running.

          • Be careful not to overload and burn it out, costing you more money than a couple extra batteries would have. }:)

            Everyone seems to read their own meaning into what I say...

            I wasn't even suggesting hooking up your car to a UPS, but hooking up the car's BATTERY, and when that runs low, recharging it with your car... No matter how much load you have comming off the UPS/inverter, it can't possibly damage your alternator.

            Cars are designed to haul themselves and your ass around, not keep your little server fa

        • I don't know about you, but if the power gets that bad off, I'm not about to rig my car up to my house. I have much better things to do than use electicity in those situations :)

          But, all in all, the alternator would be under some massive load with anything more than a computer setup and a television (*maybe*) connected to the battery. It's just too much juice... especially for some of the not so new cars that have factory alternators that really suck.
          • Of course... I'm sure nobody would want to be able to watch TV, listen to the radio, or turn on the lights.

            the alternator would be under some massive load with anything more than a computer setup and a television

            I wasn't suggesting having a computer, radio, and TV on, I was suggesting lights, and one of those. I think that would be obvious by the "300watt inverter" statement.
            • A coleman lantern is a little cheaper in that case....

              would light the room as bright as a normal household light.

              That, along with a crank-powered radio, and your set. All part of the usual household "emergency plan".
              • A coleman lantern is a little cheaper in that case....

                Not much cheaper, doesn't allow you to run other devices, and is a lot of work for something you'll be using less than once every 5 years.

                would light the room as bright as a normal household light.

                Indeed, mine is very powerful, but it doesn't use normal gasoline, and most stores where you can buy fuel for it are going to be shut-down in a power outage.

                I like my coleman lantern very much, but I only use it for camping, because you have much more fle

                • I was gonna just skim over this, but I have to point out a few things:

                  Not much cheaper, doesn't allow you to run other devices, and is a lot of work for something you'll be using less than once every 5 years.

                  eh? I picked mine up for like 30 bucks or so, probably less considering I'm cheap. It's not alot of work to use, just screw on the propane tank (5 seconds of work), light a match, turn the handle and boom.. light. Admittedly you can't plug electrical appliances in, but then again our goal isn't t
        • Altenators in new cars aren't really good for this:

          An altenator is different from a generator in that the supplied field is electromagnetic...it depends on another voltage source. This source is supplied (initially) by the battery or residual magnetic field when you start up the car, and is varied by the regulator. When your car needs recharging, the regulator ups the current or voltage in the altenators field wire, and your car cranks out more volts and recharges the battery. When the battery is mostly
          • As I've said multiple times, I was no recomending using your alternator directly, but was recomending using your car to charge a car battery, and then hook that battery up to an inverter. Everyone should have at least 2 car batteries I hope.
      • Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :(

        You want fuel cells, not batteries.
    • Agreed (Score:3, Interesting)

      by The Tyro (247333)
      I've yet to see a datacenter that didn't have some sort of backup power... I've got backup power in my house, for pity sake.

      Anyone in the affected areas care to comment on what happened? Did you guys just exhaust your UPS capacity, or do you just have it for orderly shutdown?

      There's boku generators still floating around from all that Y2K kerfluffle... you could probably purchase some cheap failover power just about anywhere...
    • Re:backup? (Score:3, Informative)

      by evilviper (135110)
      Backbone routers were fully functional. The problem was than many, many smaller networks don't.

      I know it's too much to ask here, but I would suggest you read the story.
    • Re:backup? (Score:4, Informative)

      by ChrisCampbell47 (181542) on Tuesday August 19, 2003 @11:52PM (#6740842)
      Don't these backbone routers have backup? I was in an ISP server house in the UK which had a full backup system. In the case of a power failure, it had a UPS that kicked in for 10 seconds while the generator was booting up, which then provided power for the infrastructure of the building. I would find it hard to believe that in the USA they don't have similar systems?

      Uh, yeah, thanks, we never thought of that.

      UPS's run out of juice. Generators run out of fuel. Generators turn out to be less than perfectly maintained and fail after a couple hours. Budgets get trimmed, maintenance gets overlooked, blah blah blah it's never a perfect world. If it was, engineers would be replaced by algorithms.

      Companies that are dead serious about power reliability run generator tests every day, and when lightning is detected within miles we automatically start up all the generators and run off them. Yes, we actually go OFF THE GRID every time a thunderstorm rolls in, and in Atlanta that's many times a week. Of course, we've got millions of dollars an hour [turner.com] running through our facility so heads would roll if we weren't this paranoid.

      I believe we keep our tanks fueled up for 3 days of continuous service, and we pay a premium to guarantee that when the shit hits the fan, WE get refueled first.

      Of course, hardcore multiply-redundant (and *tested*) systems are something that elude the typical IT crew staffed by DeVry grads.

  • keep in mind (Score:2, Interesting)

    by citizen6350 (699527)
    Note: the dots only represent 5% of the actual NUMBER of routers downed. (though I bet their locations are based off of an average)
  • Woohoo Toronto (Score:2, Interesting)

    by Streiff (34269)
    All that power outage, and not a single network outage. Woohoo!
    • That means that although you can't work, you can still read slashdot! Wait, that's what you were doing anyways :P
    • Re:Woohoo Toronto (Score:2, Interesting)

      by malloc (30902)
      I have a friend who runs a small ISP here in Toronto. Today he told me that he never lost network connectivity during the blackout. He's with WorldCom and apparently because of W2K they built facilities downtown so they can last a whole month without power.

      -Malloc
    • Yeah Sympatico here did well. My servers did not have UPSes though and did go out, but they booted back online by themselves so no issue.

      The Toronto Public Library systems were out for a while though. Libraries here are still closed. Those were I suppose the only major servers that did go out.
  • Add one more server to that infographic, since I'm fairly certain a pic that large is going to get fragdotted quite speedily.

    -theGreater Pessimist.
  • i told them my dsl had been choking off and on for the last day and that my ip address had been changing on the hour. now i get to see pittsburghs own special dot on the map. it's quite comforting :).

    the good news is that they told me it should be fixed in 4 hours (that was two hours ago).
  • by PaulBartlet7 (629282) on Tuesday August 19, 2003 @07:11PM (#6739053)
    As the link says - http://www.socio.demon.co.uk/mphil/appendix1.html [Quote] The Internet is a web of several thousand computer networks that now extends to just about every region of the world and has 50 to 100 million users. The Internet of today has it's origins in a networking project called ARPAnet which was run by the Advanced Research Projects Agency, a science research body set up in 1957 by the Pentagon. (Hafner & Lyon, 1996, 19) The popular belief is that the military created the ARPAnet, the precursor of today's Internet, so that data held on Pentagon computers could survive a nuclear attack by the Soviet Union. Upon attack, data from computers at the Pentagon and other military installations could be uploaded (sent electronically) to other remote computers not affected by such an attack. [/quote] It's always been my understanding that the Internet would continue working after a Nuclear war, at least that was the plan. If this blackout had effected all of US / Canada like a Nuclear attack would, would any of the Net worked ?
    • Define work? With current routing topologies you take down all the tier ones and your not getting out of the USA and will have trouble getting much farther than that. Contract wise the tier ones have been applying a lot of presure on the tier 2 guys not to advertise interconnects and often have good reasons not to. Add to this the massive ammounts of long haul centralization take out a few MAE points and things would be bad VPN's are replaceing the long haul circut and as it gets nastier and naster out t
      • With current routing topologies you take down all the tier ones and your not getting out of the USA and will have trouble getting much farther than that. Contract wise the tier ones have been applying a lot of presure on the tier 2 guys not to advertise interconnects and often have good reasons not to.

        Silly question: Will IPv6 make the Internet more stable? Will it allow the tier 2 providers to trade routes more easily?
        • Silly question: Will IPv6 make the Internet more stable? Will it allow the tier 2 providers to trade routes more easily?

          I think his point is the decision is political, not technical. It doesn't matter if IPv6 makes it easier, because the Tier 1 guys don't want it to happen. But, you know, privatized telecom is good for us, ain't it?
    • computers could survive a nuclear attack by the Soviet Union.

      Well, yeah, the computers survived but the power grid that runs them and their environmental support got hosed.

    • by Ralph Wiggam (22354) * on Tuesday August 19, 2003 @07:37PM (#6739228) Homepage
      The popular belief that ARPAnet was designed to survive a nuclear war was created by a Time magazine writer who didn't know what he was talking about. ARPAnet was created so that people doing military research could share thier work and the DOD wouldn't have to pay for the same research twice. That's why the first nodes were universities and not military bases. My alma mater, Univ. of Illinois, was supposed to get one of the first nodes outside of CA, but hippie protestors delayed it for a while. Fucking hippies.

      -B

    • by SmallFurryCreature (593017) on Tuesday August 19, 2003 @08:11PM (#6739479) Journal
      You might be suprised to learn this, but army bases tend to have their own power generators. Silly but for some reason they object to being depended for power on an installation hundreds of miles away.

      So yes the original setup of the internet was to survive stuff like this. As indeed it did. In areas not nuked it continued to work just fine.

      The entire point after all was for the network to survive even if a big hole was punched into it. We just saw that happen. And talked about on the net while it happened showing that the bits around the hole kept working.


    • If this blackout had effected all of US / Canada like a Nuclear attack would, would any of the Net worked ?

      If a tree falls in the forest ...
  • We don't get to see ask.slashdot.org fall off the Internet from this morning.
  • redundancy (Score:5, Interesting)

    by paradesign (561561) on Tuesday August 19, 2003 @07:21PM (#6739115) Homepage
    Im in detroit, and after power went out, so did the network. I thought that the internet was supposed to be redundant (like power)! i thought telcos invested bigtime to keep in the 9's. its good to know it didnt do squat. at work today, we finaly got a stable connection from our upstream routers. our tech department was furious because all of our stuff was on backup, why wasnt theres?

    oh yeah, the cell network here was down for a good while after the lights went out. well not down, just full. i thought they learned on sept 11, that there wasnt enough capacity on the cell networks. but you know, i could be wrong.

    • after power went out, so did the network. I thought that the internet was supposed to be redundant

      What went out, where, when? Just because your cable modem couldn't connect, doesn't mean the internet was out ;-). I suspect you had something better than that to test, but what?

      Maybe your ISP/hosting provider is just an economy service, and doesn't have any long-term power supply.

      Although, I really don't know why so few had power... A car/truck is a 12-volt generator, and an uninteruptable power supply

    • I thought that the internet was supposed to be redundant (like power)... i thought they learned on sept 11, that there wasnt enough capacity on the cell networks

      We had a presentation last night from Tom O'Rourke [pbs.org] on the critical infrastructure affected by Sept 11 2001.

      The main point to come out of it is that most critical infrastructure is commercially run. They are designed to run and handle typical peak loading.

      And there is a difference between redundancy and surge capacity. Redundancy allows you to co
    • our tech department was furious because all of our stuff was on backup, why wasnt theres?

      if you're complaining about it, you should consider getting a second upstream so that your network doesnt have a single point of failure :)
  • Cool, Ottawa stayed up, too bad I lost my power so I couldn't enjoy it.
    • Yup, we did not fail. We do not seem to have any provincial borders either.
  • "Providing real-time internet connectivity monitoring and reporting worldwide"

    Wouldn't "real-time...monitoring" = DoS?
  • by akorvemaker (617072) * on Tuesday August 19, 2003 @07:27PM (#6739157) Homepage
    Woo hoo! Notice how all the outages are south of the border? We are so good! ;-)
  • by psylent (638032) <psylent AT natural DOT mailshell DOT com> on Tuesday August 19, 2003 @07:27PM (#6739159)
    Would love to see an animation of a webserver being slashdotted.
    • by tmu (107089)
      And yet, the webserver's fine (famous last words).

      Did y'all notice *how* *small* that animation is? Someone else here put it together. I didn't know that animated gifs could compress that much.
  • Florida??? (Score:3, Interesting)

    by advocate_one (662832) on Tuesday August 19, 2003 @07:28PM (#6739162)
    Anyone else notice that node in Florida go down as well???
    • Old Nodes (Score:3, Funny)

      by spineboy (22918)
      Those were all the old nodes that retired and moved south to Florida.......

      Oklahomah had a node go down too..WTF? Ya got me on that one...Can't even make a joke about that..
    • Yeah, I thought that was strange too. As a poster below noted, don't forget that the animation was a representative sample of only 5% of the total outages. Which doesn't go any further in explaining it, but it stands to reason that other nodes in Florida (or general vicinity) went out as well, though obviously not to the same extent as in the Northeast US.
  • i'm up in canada and when the power went out our internet conenction didn't. i know this because one of the guys from our offices out west called and asked me what was going on. tho our network did go down due to our ups running out of juice.

    the only thing that did NOT go down during the power outage up here was our telephone system. i'm not sure about our other net conenctions tho, i know my home cable connection went down, that's cause cable was out in certain areas, tho in other areas the cable was f
  • I work for IBM and our building had an hours worth of generating power left.

    45 minutes after the power went out we were still surfing the web listing to what we could from winamp radio stations, our phones are standard call center VOIP lucent stuff.. they worked fine i was calling friends in ottawa (im in toronto) letting them know what cnn.com said.

    nice to know that if the power goes out i have to walk down 11 flights of stairs, but i can still work (in the dark, no AC) on my computer and use my phone..
    • ice to know that if the power goes out i have to walk down 11 flights of stairs, but i can still work (in the dark, no AC) on my computer and use my phone.. gee.. i guess i know where my employeres priorities are.

      Should be obvious, no?
  • by loconet (415875) on Tuesday August 19, 2003 @08:13PM (#6739486) Homepage
    14th 4:20pm - Power goes out, our building's generators for some reason dont kick in.
    14th 4:35 - Most of us decide to call it a day and go home
    14th 4:30 - I'm in my car, I realize the blackout is bad when only 2 or 3 radio stations are working and the no traffic lights are. I know it was going to be a fun drive home - thank god i live 15 mins away.
    14th 4:45 - I hear the blackout expands to parts of the States. me: OH FUCK@#$@#$
    14th 4:50 - My sister sms's me on my cell telling me her and my mom are stuck in the subway - they need help. Like I care, I have my own problems , traffic is a mess and there are hundreds of psychos out.
    14th 5:00 - I get home, look for a battery powered radio and listen in.
    14th 5:30 - I get a call from my sister - they're stuck somewhere downtown. I just wish them luck.
    14th 7:00 - I realize there are no candles in the freaking house, time to look for those puppies.
    14th 8:00 - Sister & mom arrive home. me: LOL
    14th 11:30 - I go to bed & pray the blackout lasts until the middle of the next day, that way I get an extended weekend -wohooo, back to bed
    15th 7:30am - Wake up, lights are still out - no work, home free! wohoo, back to bed.
    15th 11:30 - Receive a call from my boss, asking me where the fuck i was since they got power at the office but there is a lil "issue".
    15th 12:30m - Got to the office, problem: No Internet connection, seems one of the ISP's switches went bye bye after the black out. Our main app server is down. No power you think? Nope The colo company hasnt been paying their bills and WorldCom used the blackout to pull the plug on them. Server is up and running but no outside world connection. FUCK@#*$.
    15th 1:00pm - We think , no biggy let's use one of our other servers and restore apps and data from backup. HA! yah right - Turns out a DNS servers for our backup machines had died and the backup script had stopped working 10 days earlier. Great.
    15th 2:30 - We wait to see if Worldcom is nice enough to plug the box back in.
    15th 4:30 - Yah, it ain't happening - 20+ clients are without website and apps.
    15th 5:00pm - Boss and I drive downtown to the WorldCom building to download data physically off the Box.
    15th 5:30 - Stop for gas - HA! huge lineup.
    15th 7:30 - Get into the server room, ha! the fucking cage where our box is is locked and the key is not working. One of us climbs the cage and goes into it, runs an ethernet cable from the box to the laptop. So picture these, 4 geeks inside a server room, three sitting on the floor , one inside a cage like some wild animal. I should have brought a camera. Let the tar'in begin.
    15th 9:00 - Download is completed, our asses are sore from sitting on concrete, necks hurting, and WorldCom employee happy that we're finally leaving.
    15th 9:30 - We're downtown wondering how the fuck we're going to upload 2gig+ worth of data and source code to our spare server.
    15th 10:30 - Since there is no inet at the office and our home's cable is too fucking slow (Rogers cable sucks!) , we decide we bring out the ghetto in us. We walk up and down Yonge street asking Internet cafe's if they could lend us some bandwith!!!. Yes, you hear me right, we were begging for bandwith in internet cafes.
    15th 10:45 - We decide we're hungry, so we stop at a sushi bar. After we're done we realized it might not be a good idea to eat fish after a blackout. Fridges not working aand all. Too fuckin late.
    15th 11:00 - Found an internet cafe that will let us connect the laptop to upload.
    15th 11:30 - Realize we can't do shit since the computer is in Korean , have all Win settings locked and the guy taking care of the place has no clue.
    16th 1:30am - I'm at my boss's house uploading 2gig+ data , will take about 27 hours. Ask me if I cared about the clients at this point.
    16th 1:35 - I leave and head to my friend's place where they're having beers & bbq'in on my friend's balcony. - My weekend Begins.

    Network Blackouts? Yah they suck.
  • "I swear, it's the Northeastern power outage that caused the Omaha routers to go down."
  • by AgentPhunk (571249) on Tuesday August 19, 2003 @10:09PM (#6740274)
    Who else here (who's old enough) thought of Atari's old Missle Command when you watched that animation?

    anyone? Anyone? Bueller?

    (sigh)

  • And I was starting to wonder why I see all those "I love NY" t-shirts...

    Even here in Minnesota...
  • Sender: jm
    To: dave

    Hi Dave -- for IP.

    There's an article from Heise Security in Germany at [1], which raises
    some interesting questions about whether W32.Blaster could be to blame for
    the blackout. Some translated points are at [2] -- quote: ... it becomes a bit more likely if one considers what the authors of
    that article found out:

    - The Niagara Mohawk power grid which seemed to got overloaded first
    is owned by National Grid USA.
    - National Grid is listed as an important customer
    • In line with that, it seems that one of the first sites to go down, is right along the Michigan/Canada border. It's unclear whether it goes down before or after the power outage. Now, I'm willing to bet that 1300MW flows from the Canada side of Niagra falls into Detroit.

      And if that 1300 MW can't get there by the direct route, why then the automatic system is going to reroute it around Lake Erie, resulting in a sudden swap of direction as the current goes the other way. That's what is known to have taken
  • Perhaps... (Score:4, Funny)

    by carrier lost (222597) on Tuesday August 19, 2003 @11:46PM (#6740813) Homepage

    ...it's time to revisit AC power over TCP/IP.

    MjM

    Groovy. Gear. Mod.

  • Why Boston? (Score:4, Interesting)

    by cgleba (521624) on Wednesday August 20, 2003 @12:43AM (#6741126)
    Why was there an outage in Boston, MA and Springfield, MA -- Massachusetts did not loose any power?!
  • by Radical Rad (138892) on Wednesday August 20, 2003 @12:59AM (#6741208) Homepage
    It is interesting to see how localized some of the outage was--networks in New York state right up to the Vermont border go dark while everything on the other side of the border is quiet. New York City obviously gets clobbered."

    Here is a nice satellite pic comparison of the Northeast before and during the outage from Natural Hazards [nasa.gov]

Reality must take precedence over public relations, for Mother Nature cannot be fooled. -- R.P. Feynman

Working...