Network Blackout

An anonymous reader writes "Renesys put together a special report on the effects of the recent blackout on routing and network reachability on the Internet. It includes a cool animation of networks dropping off the internet (presumably as a result of the power outage). It is interesting to see how localized some of the outage was--networks in New York state right up to the Vermont border go dark while everything on the other side of the border is quiet. New York City obviously gets clobbered."

backup?

[killermal]

Don't these backbone routers have backup? I was in an ISP server house in the UK which had a full backup system. In the case of a power failure, it had a UPS that kicked in for 10 seconds while the generator was booting up, which then provided power for the infrastructure of the building. I would find it hard to believe that in the USA they don't have similar systems?

Re:Origins of the Internet - no power, no work ?

[four12]

computers could survive a nuclear attack by the Soviet Union.

Well, yeah, the computers survived but the power grid that runs them and their environmental support got hosed.

Re: Don't backbone routers have backup?

[RollingThunder]

Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.

Why would you want it all in batteries? Use the UPS to tide you over until you can fire up the gas/diesel generator. Those you can get pretty cheaply (well, compared to 300lbs of batteries) and are useful for other things too - such as going camping.

Re:This is why we need that Martian Nuclear PP

[Anonymous Coward]

I know you're trying to be serious/relevant, but:

Ever heard of HUMOR??

Re:Origins of the Internet - no power, no work ?

[SmallFurryCreature]

You might be suprised to learn this, but army bases tend to have their own power generators. Silly but for some reason they object to being depended for power on an installation hundreds of miles away.

So yes the original setup of the internet was to survive stuff like this. As indeed it did. In areas not nuked it continued to work just fine.

The entire point after all was for the network to survive even if a big hole was punched into it. We just saw that happen. And talked about on the net while it happened showing that the bits around the hole kept working.

Quick and Dirty LIVE UPS Recharging Ideas

[BigBlockMopar]

Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.

Yeah, same up here in Ottawa, Canada... I was awakened early on Friday morning to the sound of my servers POSTing; my power was back in under 12 hours. I was lucky. :) (Made sure to double-check that hdparm was set to spin down the drives, that and killing the A/C were my contributions to energy efficiency.)

Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.

I don't have a UPS (well, I do, I got one free, but it's broken and I haven't had time to troubleshoot it - anyone got schematics for an APC Back-UPS Pro 280?), so your mileage may vary. If the UPS runs off 12V batteries, you might be able to:

• cobble a set of binding posts onto the side of its case, in parallel with the battery, and connect them to the battery in a running car. (Essentially, "jump-start" your UPS. Start the car first.)
• Replace the 12V gel-cel battery with a good old-fashioned car battery. Even a weak used one should run it for a lot longer, but I haven't seen how charge current is regulated when the UPS is on AC, so I don't know how well the UPS's charging circuits will tolerate it.
• Scoop an old gas lawnmower out of the garbage (Briggs and Stratton or Techumseh 4-stroke motors are preferable and very reliable if you keep them well tuned). Fix it, and install a pulley where the blades were (usually a 3/4" or 1" keyed shaft, and you want to take a 4L belt of the required length). Cut a hole in the deck, install a bracket, and hang a GM 1-wire alternator (1975-1985 models) in there. The lawnmower's deck is ground, the plastic-insulated bolt on the back is the positive. Mount a car battery onto it, and you have a portable jump-starter and 12V generator. Good also to weld on a perch for your toolbox. (Built one for myself, works *great* in junkyards when you want to test compression or oil pressure in an old car.)

Note that I don't know how the UPS's inverter will handle running at rated load for longer than the internal battery is capable, nor do I expect that the UPS will have much noise suppression on the battery leads - after all, batteries themselves are pretty much noise-free electrical sources and alternators are not.

Re: Don't backbone routers have backup?

[Phroggy]

Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :(

You want fuel cells, not batteries.

Re:Map of UPS battery exhaustions

[LauraW]

At a company I used to work for the network operations folks were supposedly bragging about how redundant the servers were, with backup power, automatic network failover, and so on. So an executive decided to test them on it: he grabbed a power cord from one of the switches and pulled it out of the wall. Oops. It turned out that one wasn't redundant. Or rather, it technically was -- there were two of them -- but the failover mechanism didn't work.

At my last job we had similar problems. The system was heavily dependent on JMS, so there were rundandant JMS servers. Unfortunately, the first time the primary one on the production network went down under load all the client systems had to send tons of JMS messages around as part of the recovery process, which created a snowball effect that took down the secondary server and many of the other clients too. And then of course the clients started coming back up and sending out JMS messages to announce the fact.... (It turned out to be a bug in the JMS client that eventually got fixed, but it wasn't pretty while it was happening.)

Moral: if you haven't tested the redundancy / failover / power failure mechanism, it might as well not exist.