Network Blackout 183
An anonymous reader writes "Renesys put together a special report on the effects of the recent blackout on routing and network reachability on the Internet. It includes a cool animation of networks dropping off the internet (presumably as a result of the power outage). It is interesting to see how localized some of the outage was--networks in New York state right up to the Vermont border go dark while everything on the other side of the border is quiet. New York City obviously gets clobbered."
keep in mind (Score:2, Interesting)
Woohoo Toronto (Score:2, Interesting)
Re: Don't backbone routers have backup? (Score:5, Interesting)
Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.
Agreed (Score:3, Interesting)
Anyone in the affected areas care to comment on what happened? Did you guys just exhaust your UPS capacity, or do you just have it for orderly shutdown?
There's boku generators still floating around from all that Y2K kerfluffle... you could probably purchase some cheap failover power just about anywhere...
redundancy (Score:5, Interesting)
oh yeah, the cell network here was down for a good while after the lights went out. well not down, just full. i thought they learned on sept 11, that there wasnt enough capacity on the cell networks. but you know, i could be wrong.
Florida??? (Score:3, Interesting)
Re:Woohoo Toronto (Score:2, Interesting)
-Malloc
Map of UPS battery exhaustions (Score:4, Interesting)
You don't have backup power on your home LAN? Pffft, and you call yourself a geek.
From what I know, this failure happened from New York City to Ohio in a matter of under a minute. So, I guess the dots are more representative of the average lifespan of the (formerly) fully-charged batteries in one's UPS.
As for me, I was dead in the water. At home, it was instantaneous (I'm too cheap to buy a UPS for a site which is just for my personal amusement); at work, it was 10 minutes of standing there in the server room listening to the frantic beep-beep-beep of UPSes all around me, and then rushing around to connect low-power LCD monitors to the servers that someone else forgot to connect to the UPS's shutdown signals... (of course, this after the realization that the emergency generator is running, but not actually even connected to the servers... [grumble])
speaking of network blackouts (Score:5, Interesting)
14th 4:35 - Most of us decide to call it a day and go home
14th 4:30 - I'm in my car, I realize the blackout is bad when only 2 or 3 radio stations are working and the no traffic lights are. I know it was going to be a fun drive home - thank god i live 15 mins away.
14th 4:45 - I hear the blackout expands to parts of the States. me: OH FUCK@#$@#$
14th 4:50 - My sister sms's me on my cell telling me her and my mom are stuck in the subway - they need help. Like I care, I have my own problems , traffic is a mess and there are hundreds of psychos out.
14th 5:00 - I get home, look for a battery powered radio and listen in.
14th 5:30 - I get a call from my sister - they're stuck somewhere downtown. I just wish them luck.
14th 7:00 - I realize there are no candles in the freaking house, time to look for those puppies.
14th 8:00 - Sister & mom arrive home. me: LOL
14th 11:30 - I go to bed & pray the blackout lasts until the middle of the next day, that way I get an extended weekend -wohooo, back to bed
15th 7:30am - Wake up, lights are still out - no work, home free! wohoo, back to bed.
15th 11:30 - Receive a call from my boss, asking me where the fuck i was since they got power at the office but there is a lil "issue".
15th 12:30m - Got to the office, problem: No Internet connection, seems one of the ISP's switches went bye bye after the black out. Our main app server is down. No power you think? Nope The colo company hasnt been paying their bills and WorldCom used the blackout to pull the plug on them. Server is up and running but no outside world connection. FUCK@#*$.
15th 1:00pm - We think , no biggy let's use one of our other servers and restore apps and data from backup. HA! yah right - Turns out a DNS servers for our backup machines had died and the backup script had stopped working 10 days earlier. Great.
15th 2:30 - We wait to see if Worldcom is nice enough to plug the box back in.
15th 4:30 - Yah, it ain't happening - 20+ clients are without website and apps.
15th 5:00pm - Boss and I drive downtown to the WorldCom building to download data physically off the Box.
15th 5:30 - Stop for gas - HA! huge lineup.
15th 7:30 - Get into the server room, ha! the fucking cage where our box is is locked and the key is not working. One of us climbs the cage and goes into it, runs an ethernet cable from the box to the laptop. So picture these, 4 geeks inside a server room, three sitting on the floor , one inside a cage like some wild animal. I should have brought a camera. Let the tar'in begin.
15th 9:00 - Download is completed, our asses are sore from sitting on concrete, necks hurting, and WorldCom employee happy that we're finally leaving.
15th 9:30 - We're downtown wondering how the fuck we're going to upload 2gig+ worth of data and source code to our spare server.
15th 10:30 - Since there is no inet at the office and our home's cable is too fucking slow (Rogers cable sucks!) , we decide we bring out the ghetto in us. We walk up and down Yonge street asking Internet cafe's if they could lend us some bandwith!!!. Yes, you hear me right, we were begging for bandwith in internet cafes.
15th 10:45 - We decide we're hungry, so we stop at a sushi bar. After we're done we realized it might not be a good idea to eat fish after a blackout. Fridges not working aand all. Too fuckin late.
15th 11:00 - Found an internet cafe that will let us connect the laptop to upload.
15th 11:30 - Realize we can't do shit since the computer is in Korean , have all Win settings locked and the guy taking care of the place has no clue.
16th 1:30am - I'm at my boss's house uploading 2gig+ data , will take about 27 hours. Ask me if I cared about the clients at this point.
16th 1:35 - I leave and head to my friend's place where they're having beers & bbq'in on my friend's balcony. - My weekend Begins.
Network Blackouts? Yah they suck.
KVM Switch Loathing, IIS in Ironic Places (Score:5, Interesting)
You've got to be kidding....
No.
I hope that it was someone elses negligence & not yours!Yes. In fact, I did learn something: there's a very good reason that I hate KVM switches.
Solution (probably falling on deaf ears, but I shall try anyway):
- Keyboard and monitor on all servers. Monitors should be LCD to save UPS batteries. This way, if the UPS shutdown sequence fails, it can be done manually. (A distressing number of these machines run Redmond's cancer, so having one central "shutdown console" is less practical.)
- Serial shutdown outputs from UPS will be "broadcast" to all hosts connected to that UPS. Will require making a custom cable with a few MAX232 line-driver ICs and hang it off a machine's PS/2 port for power.
- Backup generator is rated for 90kW and showed only about 10kW load running emergency lights around the building. Time to tap out that extra 80kW of capacity. Why this wasn't checked before is an absolute mystery to me.
- Reorganize server interconnections. The network switch for the users' LAN wasn't on UPS (but neither are the users, so it didn't seem like a big deal). However, over the years, some stuff has come to rely on mapped drives... fortunately in this case, we're not so lucky to have hard-mounted NFS anywhere.
:)
- Have occasional practice power outages before long weekends. Who doesn't test their UPS by unplugging it from time to time? [grin]
I guess I'll stop complaining about the 50 l-users who ignored my emails on Jul.28 - Aug.1 warning them of the impending RPC vulnerability worm that would destroy their data.... (well, it didn't destory their data, but they did ignore my warnings & they did get the worm!)Heheh... Yeah, I know the type.
Along those lines, how's this for bitter irony? The Canadian government's "Office of Critical Infrastructure Protection and Emergency Preparedness" [ocipep-bpiepc.gc.ca] runs IIS on Windows 2000. (Note also the subjects of a couple of the bulletins on their site...) Somehow, this reminds me of doctors who smoke, or mechanics who don't change their oil.
My tax dollars at work. [sigh]
Re:Canada did not fail (Score:2, Interesting)
The line is actually right beside my house. The block on the left was dark, humid, black Ontario power. On the right (and in my house)... bright, cool, soothing Quebec power.
So I strung up a couple hundred feet of electrical cable, pulled out the coolers, invited over all the neighbours, and drank cold beer all night.
On another note.. did anyone else notice how many more of the stars they could see that night?!?
Re:For the "Deregulation was the problem" nutjobs (Score:3, Interesting)
Interesting story. CNN and a bunch of people on NANOG (www.nanog.org) were speculating that this outage was caused by the msblaster worm. This story backs up at least the feasibility of that.
W32.Blaster the cause of the blackout? (Score:2, Interesting)
To: dave
Hi Dave -- for IP.
There's an article from Heise Security in Germany at [1], which raises
some interesting questions about whether W32.Blaster could be to blame for
the blackout. Some translated points are at [2] -- quote:
that article found out:
- The Niagara Mohawk power grid which seemed to got overloaded first
is owned by National Grid USA.
- National Grid is listed as an important customer of Northern
Dynamic who call themselves the "OPC Experts".
- OPC is an acronym for OLE for Process Control and is used for
communications between control systems.
- OPC is based on DCOM, exactly that Windows technology attacked by
W32.Blaster.
- One symptom of a W32.Blaster attack is that a crashing DCOM
service (not only under Windows), often taking down the whole
server.
One usage of OPC is the coupling of so-called SCADA (Supervisory Control
and Data Acquisition) systems. Among other things is SCADA used in power
plants and grids to exchange data between some central instance and
external measuring units. And for some reason did the monitoring system
which should prevent snowball effects like the one on thursday from
happening.
So the questions the authors of the article have are:
- For which processes does National Grid utilise OPC?
- Were there any problems regarding OPC when the power went down?
- If yes, were they related to W32.Blaster?
1. http://www.heise.de/newsticker/data/ju-15.08.03-0
2. http://msquadrat.de/archive/03/08/16/02
--j.
Why Boston? (Score:4, Interesting)
Satellite View of the Outage (Score:3, Interesting)
Here is a nice satellite pic comparison of the Northeast before and during the outage from Natural Hazards [nasa.gov]