Electricity Outage Puts Routing to a Tough Test 233
infofarmer writes "Today at about 11:30 MSD (GMT+4) a major electricity outage in Moscow, Russia brought new meanings to words like "uninterruptible", "redundant" and "uptime" for network administrators, who haven't experienced such harsh and unexpected power failures since the USSR got its Internet connection. Half of the city is totally out of electricity - including subway and the most important traffic exchange point, half of the top russian sites went down, including www.mail.ru, www.rambler.ru, www.lenta.ru, some of them haven't been brought up yet. IP packets going from ADSL users in Moscow to some local sites got rerouted to somewhere in London and then back to Scandinavia, where they met their "No route to host" deadend. Other routers found themselves in a loopback, which made many packets get dropped with TTL expired. The point is that most of popular servers have got two or three mainline Internet connections, but lack of BGP/RIP2/whatever configuration resulted in packets losing their way to hosts."
Probably unrelated (Score:5, Funny)
Re:Probably unrelated (Score:4, Insightful)
In all seriousness, we (or, y'know, lawmakers somewhere) should really look for the spam volume trending before-during-and-after the outage.
A surprise for some, no surprise for the rest of us?
In all seriousness... (Score:5, Informative)
I've gotten none in the last couple hours.
Now we know ... (Score:2)
Re:No spam for 4 hours! (Score:5, Funny)
Re:No spam for 4 hours! (Score:2)
Re:No spam for 4 hours! (Score:2)
Re:In all seriousness... (Score:2)
I too delete mail from spam folder on every occasion. My reason is to catch false positives as I've had some (russian+finnish mails). Removing spam regularly is only way to keep up with 'seen those' -level. (You could 'select all' and 'mark read', but with same work one could just delete them, which I do.)
Re:Probably unrelated (Score:3, Funny)
I'm surprized that nobody put some (Score:2)
Re:Probably unrelated (Score:3, Informative)
So... no prizes for guessing where the control machines for the botnet were.
Re:Probably unrelated (Score:2)
LOL (Score:4, Funny)
Can someone give this guy a metal?
Re:LOL (Score:5, Funny)
Re:LOL (Score:2)
Re:LOL (Score:2)
Re:LOL (Score:2)
In soviet russia... (Score:5, Funny)
Re:In soviet russia... (Score:5, Funny)
Re:In soviet russia... (Score:4, Insightful)
Re:In soviet russia... (Score:2)
Re:In soviet russia... (Score:2)
no more all off mp3 .com (Score:5, Funny)
Obviously the MPAA/RIAA are to blame..
Re:no more all off mp3 .com (Score:3, Informative)
silly packets (Score:2, Funny)
Luckily, (Score:4, Funny)
Re:Luckily, (Score:2)
Re:Luckily, (Score:4, Funny)
Re:Luckily, (Score:2)
Somewhere, an RIAA executive is crying.
Re:Luckily, (Score:4, Interesting)
I have a mirror... (Score:4, Funny)
Re:I have a mirror... (Score:2)
Wow (Score:3, Funny)
Alternate Headline (Score:4, Funny)
Correct router configuration can be difficult!
I'm worse than Russia. (Score:3, Funny)
Russia's connections at least made a couple of hops before dying. Mine died on 1 hop. It did illustrate the uselessness of a battery-backed up modem on my network, however.
Re:I'm worse than Russia. (Score:2)
*I've got a laptop, so yes, I do still have a functioning computer in that situation
Re:I'm worse than Russia. (Score:2)
Does the switching station keep the other side of your DSL connection and their routers on UPS?
Re:I'm worse than Russia. (Score:2)
Re:I'm worse than Russia. (Score:2)
Re:I'm worse than Russia. (Score:2)
given all that would it really be worth doing?
Re:I'm worse than Russia. (Score:2)
Re:I'm worse than Russia. (Score:3, Interesting)
If you had any servers running, though... well... time to get a UPS.
That is obviously an option, but that laptop is set up only with wireless and I really didn't care enough to configure the landline and assign IPs.
My server has gone down because of the power twice in the past 2 days. I would consider UPS if it mattered.
its quite normal (Score:2)
many systems are set up to use an upstream transit link if no local routes can be found so packets can get sent quite a long way before they finally hit a brick wall.
See what happens (Score:2)
Yes, but... (Score:5, Funny)
Re:Yes, but... (Score:2)
Re:Yes, but... (Score:2)
The submitter has to have his priorities checked (Score:4, Interesting)
Nice choice of words (Score:2, Funny)
and
who gives a shit about Internet?
What exactly are you trying to say here?
in times like these, the 'net is a godsend (Score:5, Informative)
Re:in times like these, the 'net is a godsend (Score:2)
Insofar as all the geeks in the world need to know, the headline could also have been that the power was out in Moscow, not that the internet was.
Re:The submitter has to have his priorities checke (Score:5, Insightful)
There's more traffic on the 'net than pr0n, wazrez, mpEs and /.
Some of it actually matters.
Re:The submitter has to have his priorities checke (Score:3, Insightful)
The same can be said of the electrical grid. And the cellular network. And the water network. And the sewage system. Or the public road infrastructure. Or the food distribution chain. Face it - virtually every aspect of modern life requires you to rely almost completely on infrastructures that you do not own.
One has to remember what the internet actually i
Re:The submitter has to have his priorities checke (Score:2)
Re:The submitter has to have his priorities checke (Score:5, Interesting)
This is what is supposed to happen. All (nearly all?) sewage treatment plants have a bypass to send the input straight to the output, which is usually a river or lake.
They do it because when a treatment plant cannot accept any more sewage, whether due to excessive water input by rain, or by power loss, the customers are better served by *NOT* letting the sewage back up into their houses. The stuff has to go *somewhere* when all their holding tanks are full. This is the last-resort method of dealing with problems at such plants.
Re:The submitter has to have his priorities checke (Score:2)
you are right nowhere near as bad as letting it back up through the sewer systems into populated areas but that doesn't mean its a good thing just less bad than the availible alternatives.
those sewage treatment plants were installed for a reason you know.
Re:The submitter has to have his priorities checke (Score:5, Insightful)
Re:The submitter has to have his priorities checke (Score:2, Insightful)
The reason this is on ther front page is because the internet is suposed to be able to handle things like this. People will be watching how the routers automatically deal with the outage (there's one response like that already), and what manual intervention it needed. Hopefully this information will be used for training the next generation of router admins.
Even if this was because of a meteor
Re:The submitter has to have his priorities checke (Score:2)
Odd.... (Score:5, Funny)
Wonder if these are somehow related.
Re:Odd.... (Score:2)
Obviously in need of a tighter setup (Score:4, Funny)
Those mischevious packets. Unraveling networks where'er they roam.
Perfect opportunity... (Score:4, Funny)
Eveything is fine... (Score:2, Redundant)
No interuption of service from where I stand
Re:Eveything is fine... (Score:2)
Re:Eveything is fine... (Score:2)
allofmp3.com OK (Score:2)
In other news... (Score:3, Funny)
Not big headline? (Score:3, Interesting)
Re:Not big headline? (Score:3, Insightful)
Well, first off, you're factually incorrect. The outage 2 years ago affected a large area of the Eastern United States as well as some areas of Canada, not just NYC. Furthermore, the sources you cite are all american sources. It's no surprise that they tend to report american events more than world events.
Re:Not big headline? (Score:2)
Re:Not big headline? (Score:2)
Some coutries are more inclined to get the meaning of international then others. Here in Brazil international mostly means USA and Europe, we don't even get to know what happens in Argentina or Chile, if it does not involve Brazil in some manner.
Internet... works! (Score:5, Informative)
But then routes began to appear again! I was amazed, Internet routed itself around damaged segments, packets were routed through Japan (!), Finland and Holland instead of Moscow. The most funny part was when I traced the route to a computer in the next building - it went through Saint-Petersburg
I was able to access Slashdot, and most of Russian sites (http://newsru.com/ [newsru.com] , http://ntv.ru/ [ntv.ru] , http://nbc.ru/ [nbc.ru] not directly affected by outage.
Re:Internet... works! (Score:5, Interesting)
That's why.
TCP/IP and the Internet anticipate cooperation among sites. You and your neighbors should all happily route each other's packets.
The trouble is that in many places it doesn't work that way. There are rural "leaf" nodes, of course, but there are many more sites which have only one connection because of what I consider to be petty business decisions.
Two competing ISPs in the same area should share a direct link to each other. If they have different upstream providers, then when one provider goes down the other picks up the slack. In any case local traffic should stay local.
The fear, of course, is that one ISP will choose a bad provider and take advantage of the other. That has an easy fix: if the other one starts to abuse you, pull the plug.
Single points of failure are not supposed to exist.
Re:Internet... works! (Score:3, Funny)
Yeah, but Slashdot doesn't work anyway.
Spam (Score:2, Funny)
Re:Spam (Score:2)
I also noticed my
Maybe if we can knock down the routers in China, it will be as quiet as the skies on 9/11.
Routing (Score:3, Interesting)
Re:Routing (Score:3, Informative)
Right now poor admins are trying to find stable routes for Russian traffic, which overloaded some international channels.
Fzzzp (Score:3, Funny)
Off-topic, but an interesting read
UES Management Faces Criminal Investigation (Score:4, Informative)
From the article:
Russian prosecutors on Wednesday opened a criminal case against the management of power monopoly Unified Energy System (UES) after a major power outage in Moscow, agencies reported Wednesday.
The case was opened to investigate possible negligence, the Interfax agency quoted the Prosecutor General's Office as saying.
And in other news... (Score:2)
And in other news..phishing sites have decreased by 50%.
no backup power?! (Score:2)
or is there more to this story than we are being told right now?
Re:no backup power?! (Score:2)
Also, remember that even with those precautions, a failure in any UPS or generator anywhere between the user and the servers can result in an outage. That includes telco equipment in other buildings as well.
There have been several high profile datacenter power outages in big U
Re:no backup power?! (Score:2)
do major datacenters in russia not bother with backup power?
are thier backup power systems in such a state of disrepair that a large proportion of them simply didn't work?
or whats going on?
Oh damn (Score:2)
The big question is... (Score:2)
I think this is a "political outage" of some sort (Score:5, Interesting)
While the guy is not as powerful as he was a few years ago, he still poses a significant threat to Putin's third (and fourth, and so on) term presidency, and further concentration of power in Putin's hands.
So within half a few hours of outage, Putin blamed Chubais directly for this, and Russian justice dept opened up a criminal case against him. If you know anything about Russia, you know that Russian DOJ (Prokuratura) doesn't start criminal cases against wealthy and powerfull businessmen and politicians unless instructed to do so by Putin.
So I'd bet dollars against donuts that this outage was caused by folks from Lubyanka (FSB aka KGB) purely to remove Chubais, and if cards play well maybe even give him a lengthy prison term.
let's separate that in two issues (Score:2, Insightful)
Whether the FSB caused the outage directly, to prompt an attack on Chubais is another matter. Maybe they were working on a plan but it wasn't ready yet. They have a lot to do
Even Putin sometimes just exploits opportunities.
In any case, the outcome is the same.
One thing is what he "states", and another is what (Score:2)
Consider the "dictature of law" doctrine that he so vehemently supported when he came into office. For some reason, dictature of law applies only to his (or his buddy oligrachs') political opponents. Khodorkovsky was an opponent, Chubais is an opponent, Berezovsky, Gusinsky, you can continue this list yourself. Abramovich, Deripaska, Mamut, et
Frozen in time... (Score:2)
I suppose that was taken from when all the digit clocks stopped working and just displayed one time...
lack of BGP/RIP2/whatever configuration (Score:5, Informative)
UPS: at least in one place in MSK-IX they did have proper UPS backups, you can tell from routing tables that some BGP connections have an uptime of 4 weeks plus. They did bounce (or it had a power failure) one of their core routers as all those peering connections only have an uptime of 8.5 hours. I'd rather not provide a link to this as the last thing they need is their core routers slashdotted with BGP table summary requests.
Connectivity: it appears MSK-IX is peered with at least 12 other sites that are also peered with another major IX. For example they are connected to three other sites that are also connected to AMS-IX and four other sites that are also peered with LINX, among a few others with only 1 connection to another Internet Exchange. Many of these were thru Informtelecom XXI, so if they also had power problems everything was running on 50% normal capacity. There should have been enough connections to keep things running (i.e. no single point of failure), but that is assuming everything is working/powered, and assuming these guys in the middle could/would handle all the traffic (unlikely).
BTW, packets don't lose thir way, routers lose their routes to destinations. When all the crap started the routes began to "flap", i.e. go up and down as routers were reset, power came back on, routers went back down under the heavy load, manually trying to route around the problem, etc. When your peer sees your routes flapping, they usually put a holddown on them for a period of time, meaning they won't readvertise your route updates to other routers on the internet (said flaps propogate all over the world, putting undue stress on other routers). So even once you get everything working again, the internet waits for a little bit to accept your routes. Well, some do and some don't or some wait longer. That's why you see routers still forwarding packets to London, apparently London thinks it can still get to Moscow so it's still advertising routes. You don't get the count to infinity problem with BGP, but loops are still possible, especially during major outages and route flapping. And routers get "routing loops," not "found themselves in a loopback."
I provided as much details as I could, it's lacking in a few places because I can't follow russian websites.
RIP!?!?! (Score:2)
CIA running Netwar Wargame This Week (Score:3, Interesting)
http://apnews.myway.com/article/20050525/D8AAFUIO
Am I just blowing smoke here...?
Re:In soviet russia... (Score:2)
Re:In soviet russia... (Score:2)
Re:umm its not the USSR :) (Score:2)
Re:umm its not the USSR :) (Score:2)
At the time that the geographic region was hooked to the internet originally, it was called the USSR; this statement is technically correct.
Re:umm its not the USSR :) (Score:2)
Hey, are you the guy with the "GET A BRAIN! MORANS" sign? [about.com]
Re:Likely consequence (Score:2)
The Patriot Act doesn't go nearly far enough.
Just wait until we pass the
Securing America via Federal Enforcement (SAFE) Act and the
National Operations againt Criminal Racketeering on the Internet and in Multi-user Environments (NO CRIME) Act.
It's coming boys and girls, it's coming.