Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
The Internet

LiveJournal Blackout Analysis Online 333

Posted by CmdrTaco from the when-it-all-hits-the-fan dept.
Hakubi_Washu writes "LiveJournal has posted their official analysis of what happened last Friday. Apparently someone "accidentally" pushed the emergency power off (which should keep all power off, even UPS), reset it and ran off. They had problems to come back up fast, because of "9 machines with faulty motherboards with embedded NICs that don't do auto-negotiation properly", Machines not fully rebooting for analysis reasons and few others. "
This discussion has been archived. No new comments can be posted.

LiveJournal Blackout Analysis Online More

LiveJournal Blackout Analysis Online

Comments Filter:

  • Lesser OS... (Score:5, Funny)

    by Anonymous Coward on Thursday January 20, 2005 @03:28PM (#11423679)

    They should be using OpenBSD. It can run right through power failures [grub.net]

  • The less we've learned... (Score:5, Funny)

    by geoffspear ( 692508 ) * on Thursday January 20, 2005 @03:28PM (#11423686) Homepage
    Don't let your clients near the Big Red Button without an escort. Preferably an armed one.

  • faulty mobo's (Score:5, Interesting)

    by Lifthrasir ( 646067 ) on Thursday January 20, 2005 @03:29PM (#11423697)
    so, they had faulty motherboards, knew about it, and didn't do anything to fix it before they had a major outage?
    • The solution is even funnier...
      To get them back up they need somebody at the NOC to plug them into a compatible switch, let them autonego, then switch them to their real switch.
      This is how a company with Millions of paying accounts runs its data center, and they even knew about the problem!
    • Maybe faulty, maybe not. There are a lot of incompatibilities and general "flakiness" with some network auto-negotion interactions. It's a fairly standard precaution in large network environments that servers should not rely on auto-negotiate and instead should have their speed and duplex settings hard-coded.

      In reality, the only places where auto-negotiation is important are mobile devices (laptops) which may connect to a variety of network connection types or for the home user "plug-and-play" market. Ma
  • Now, if slashdot could fix their servers, so we wouldnt get thoose annoying 503 sites..
    I havent seen them that much lately, but then i havent been online that much either...
    • Now, if slashdot could fix their servers, so we wouldnt get thoose annoying 503 sites..

      You get 503 sites? I only reach one at slashdot.org

      Then again, you're a subscriber. Who knows what goodies you lucky few get here...

  • Oppsie (Score:5, Funny)

    by darkstar949 ( 697933 ) on Thursday January 20, 2005 @03:29PM (#11423705)
    "I'll just set my coffee down here, and..."
    ...
    "Oppsie, I hope that button wasn't anything important."

  • History Eraser Button (Score:5, Funny)

    by bsd4me ( 759597 ) on Thursday January 20, 2005 @03:30PM (#11423706)

    Ah, the famous History Eraser Button rears its ugly head. I think that everyone who has worked in a large datacenter or lab environment with one of these has a story to tell...

    • Re:History Eraser Button (Score:5, Interesting)

      by scribblej ( 195445 ) on Thursday January 20, 2005 @04:06PM (#11424157)
      I'll go right ahead then. I was consulting for State Farm installing machines that were supposed to help with the Y2K problem. Hell if I know, I just got the box, went to the site, installed it and made sure it was working. Easy. I had five to do a week, and would be done by Tuesday morning and helping out other contractors on similar projects.

      I'll never forget my visit to the State Farm DSO in Detroit, MI. I'd just physically installed the new machine, at the bottom of a rack, and stood up.

      Stood up putting my shoulder right into the unprotected "History Eraser Button" on the wall. The screams of the employees working int he datacenter could be heard all the way back home in Chicago, I've no doubt.

      Then it turns out the fuses which will reset the systems in the datacenter are in a locked cabinet.

      Then it turns out no one on site has a key.

      Fortunately, I found that the cabinet will pop open if you kick it hard enough. Hey, I was panicking, okay?

      And get this. After it was all over and I realized I probably wouldn't get killed by anyone... they told me "It's okay, this happens all the time. The guy installing the A/C unit last week did it too."

      Maybe they should have put a cover over the damn button then. Morons.

      • I was consulting for State Farm installing machines that were supposed to help with the Y2K problem.


        Hey! I worked that project too... it was fun, but mindnumbing. They actualy sent me to New Orleans for an install on fat tuesday.

        Mardi Gras on an expense account :)

    • A couple of years ago, when our server room was being 'certified', one of the specific checks was "No, big red button, check". One of the guys in the group came up with a story about how someone's kid at the end of a 'tour' thought that the 'big red button' was ment to be pushed.
  • /.s current poll [slashdot.org] now?

  • Fascinating read (Score:5, Insightful)

    by Saint Aardvark ( 159009 ) * on Thursday January 20, 2005 @03:31PM (#11423728) Homepage Journal
    It's amazing how much you can learn from things going horribly wrong. :-)

    Congrats to the LJ folks for getting things working, taking the time to do it right, and giving an admin's-eye-view into what actually happened.

    • Agreed. I always appreciate when people explain how large scale outages happened, were able to happen, how they fix it, and what they do to prevent it happening again. It's useful (and good for your employment status) to learn from other people mistakes rather than your own.
      So Slashdot - what are all the 500 errors about then? :)

  • Missing opportunities (Score:4, Funny)

    by Rosco P. Coltrane ( 209368 ) on Thursday January 20, 2005 @03:33PM (#11423755)
    Apparently someone "accidentally" pushed the emergency power off

    They had to power back on when they realized deadjournal.com [deadjournal.com] was already taken...

  • LJDotting: LJ user base vs Slashdot user base. (Score:5, Funny)

    by TrevorB ( 57780 ) on Thursday January 20, 2005 @03:33PM (#11423759) Homepage
    If Mr. "I Pushed The Big Red Button"'s personal information ever gets published....

    LJ's active user base is easily 10x that of Slashdot's. We'd have to come up with a new term for the internet event that pales any slashdotting that ever came before.

  • Auto-negotiation (Score:4, Informative)

    by stilwebm ( 129567 ) on Thursday January 20, 2005 @03:34PM (#11423772)
    When I first moved company servers in to a new colo four years ago, their engineers advised me that I should turn auto-negotiation off on every port, including our switches and host NICs. I asked why they recommended this and they replied, "trust us, auto-negotiation causes problems when you least expect it." I went ahead and fixed the port speeds everywhere. Now I understand why.
    • If you know what speed port you are plugging in to why would you need to autoneg?

      It's a convenience that isn't always needed.

    • Re:Auto-negotiation (Score:5, Insightful)

      by jjgm ( 663044 ) on Thursday January 20, 2005 @03:56PM (#11424030)

      Sounds like a classic Cisco problem. I don't know what switches LJ were plugged into, but for years most Cisco switches would autonegotiate 100/half-duplex if the NIC was locked to 100/full; conversely, sometimes, NICs would autonegotiate 100/half if the Cisco was locked to 100/full.

      They're cheeky enough to document this [cisco.com] now. It's a feature, not a bug! Honest!

      • The part I like is they are claiming that everyone else is wrong, and they are right. ;)

        I don't buy Cisco anymore for this very reason, it's not just their switches, it's on everything they make that has a NIC.

        I deployed some CSS's, right after Cisco bought ArrowPoint, and they did auto correctly. Another client deployed some a couple of months ago, and auto was broken. Cisco is the Borg! ;)

      • You, sir, are an idiot. (Score:5, Informative)

        by Anonymous Coward on Thursday January 20, 2005 @04:11PM (#11424235)
        Go ahead and read up on how auto-negotiation works. I'll wait...

        No, really. Go read up on it...

        Okay, since you don't bother reading up on it, and since you claim that someone's cheeky because they *document* what happens when you misconfigure a connection, I must conclude that you, sir, are indeed an idiot.

        (To summarize for those of you who won't bother to look it up, a NIC can sense the carrier for 100, so it can differentiate 10/100. Full and half are actively negotiated by the two sides of the connection. If side 'A' is hard set to 100/full, it won't negotiate with the other side. Hearing no negotiation, side 'B' will assume the NIC doesn't support full duplex connections and failover to half duplex. This is the proper, standardized, documented behavior. Anything else would require the psychic interface spec that *still* hasn't been finalized.)

  • ...and ran off? (Score:5, Funny)

    by stratjakt ( 596332 ) on Thursday January 20, 2005 @03:35PM (#11423784) Journal
    What do you mean, ran off?

    Ran off skipping and giggling, like a 13 year old who just put toothpaste on the toilet seat?

    Or do you really mean, slunk off, like my dog does when I walk in and find her curled up on top of the remains of the remotes for the TV, TiVo, DVD player and stereo?

    My dog likes remote controls more than snausages.

    OT: Anyone know where (brick and mortar) to get a replacement (original) TiVo remote?
    • Ran off skipping and giggling, like a 13 year old who just put toothpaste on the toilet seat?

      By any chance, was his name "Zero Cool"?

  • Credit (Score:5, Informative)

    by XorNand ( 517466 ) on Thursday January 20, 2005 @03:38PM (#11423819)
    Anyone who's a paid member of LJ can get a 2-week credit here [livejournal.com].
  • I must compliment LJ for at least being honest with their system... many would lie and say "it was the datacenter's fault".

    They at least admit their own systems weren't perfect... and clearly explained each fault they observed.

    Good info.

  • Ahhhh silence is GOOOOLDEN (Score:4, Funny)

    by ShatteredDream ( 636520 ) on Thursday January 20, 2005 @03:41PM (#11423853) Homepage
    *crickets chirping* That's the sound millions of teenage girls not using up bandwidth and disk space talking about boys, jcrew and high school/college drama.

  • machine failure (Score:4, Insightful)

    by br00tus ( 528477 ) on Thursday January 20, 2005 @03:41PM (#11423858)
    "They had problems to come back up fast, because of '9 machines with faulty motherboards with embedded NICs that don't do auto-negotiation properly", Machines not fully rebooting for analysis reasons and few others.'"

    I was a sysadmin at a Fortune 100 company with thousands of servers. Every Saturday evening, we rebooted all of our servers. We almost always had several machines which would not come back up for one reason or another - so we dealt with it then, on Sunday morning, instead of during the week when a reboot of a critical machine that did not work would be much worse. Scheduled reboots are a part of good systems administration. If once a week is too often, then once every two weeks, or once a month. With this much failure, I'm almost certain they never did scheduled reboots. They had two failures - their power failed, and then their lack of planning allowed for so much to go wrong a result of that.

    • Re:machine failure (Score:5, Insightful)

      by rjstanford ( 69735 ) on Thursday January 20, 2005 @03:47PM (#11423933) Homepage Journal
      One of the last steps of our standard deployment was a full hard shutdown and restore from backup. This was shceduled to happen approximately a week before bringing the machines live - after a lot of data setup had been done.

      Many customers - and internal staff - really, really got scared at that point. The thing is, if you don't trust your backups, what good are they? Its amazing what things got taken care of and found during double-checks the week before the backup/restoration test.

      Oh, and we always went with scheduled reboots as well, for very much the same reason as you mentioned. An hour a month of scheduled downtime is almost always available - usually we booted every week and had an optional downtime window on a monthly basis. And if your (talking to readers here, not parent) organization can't afford to be without a single machine for a 2-3 hour block once a month, WTF is your plan to handle a hardware failure? Prayer?

    • Re:machine failure (Score:3, Insightful)

      by gkuz ( 706134 )
      Every Saturday evening, we rebooted all of our servers

      Yeah, we had servers like that once, too. Ba-da-bing! Thanks, I'll be here all week.

      On a serious note, am I the only one here who thinks a world in which no one questions a policy like that is insane? We've had critical, and I mean critical, servers that have uptimes measured in years. But then again they run NetWare, or OS/400, or MVS, or.... ABW.

      Scheduled reboots are a part of good systems administration

      Yeah, scheduled, as part of a disaster re

      • Scheduled reboots are a part of good systems administration

        He's talking about Windows, where regular reboots are a good thing when they are planned, so you don't have regular reboots when they are NOT planned!

  • LOL! Kindof like when... (Score:5, Funny)

    by GillBates0 ( 664202 ) on Thursday January 20, 2005 @03:42PM (#11423873) Homepage Journal
    ...when I was on AOL and I hit the X and I couldn't talk to my AOL Buddies anymore.

    And I was like OMG I shut off the internets and stuff!!1!!

    And i called the AOL helpdesk and they helped turn it back on.

  • everybody was blaming Internap for screwing up and running a shoddy Datacenter, when actually Internap did everything they were supposed to correctly.
  • Apparently someone "accidentally" pushed the emergency power off (which should keep all power off, even UPS)

    This also raised the all-important "Why do we even have that button?" question.

    • Re:Also, (Score:3, Informative)

      by Scott Laird ( 2043 )
      "Why do we even have that button?" Because it's basically required by law. Covering them with a plastic cover doesn't seem to help either--Internap did that the *last* time someone hit the EPO button in this datacenter.
    • This also raised the all-important "Why do we even have that button?" question.

      Those buttons are generally maintenance devices; it's usually less of a button and more of a keyswitch though. So the guy comes in to service something, he needs to know that no power is anywhere in there, so he removes the key and keeps it in his pocket. Now he knows he's safe.
    • It's the law. It's also in the article.

      EPO, by the way, stands for Emergency Power Off and it's a national fire/electrical requirement for firefighters to be able to press these big red buttons near all exits that turn off all power in the entire data center
    • I keep forgetting that this is slashdot. I shouls bave put in my disclamer:

      Please, do not be alarmed or reply with an explanation. This is a joke. I am joking. You have been joked with.

      Sigh...
    • They're required by law to have it. It's a building code thing. Every data center I've ever been in has one.

      Also.. ""EPO, by the way, stands for Emergency Power Off and it's a national fire/electrical requirement for firefighters to be able to press these big red buttons near all exits that turn off all power in the entire data center."
  • Maybe they should use the Button of Doom [clinko.com] (USB) to lock the pcs down too...

  • "EPO, by the way, stands for Emergency Power Off and it's a national fire/electrical requirement for firefighters to be able to press these big red buttons near all exits that turn off all power in the entire data center."

    "...all our DBs have redundant power supplies. we'll be plugging one side into Internap's, and the other side into our own UPS, which itself is plugged into Internap's other power grid. that way if EPO is pressed, we'll have 1-4 minutes to do a clean shutdown. (but if we do the rest of

    • Re:Wait a second! (Score:3, Informative)

      by rah1420 ( 234198 )
      Technically, yes. I'm hoping that if LJ decides to implement such a scheme (let's call it "LEPO" for "Leisurely Emergency Power Off") that they run it past the fire marshal or the code inspectors first, who may have another opinion about how smart this idea is.

      "If it's stupid and it works, it's not stupid."

    • Re:Wait a second! (Score:3, Interesting)

      by psykocrime ( 61037 )
      Isn't that circumventing the purpose of the EPO? If there's a smokey fire in there and the firefighters have to enter the room and start spraying water around, won't a few machines glowing for four minutes after the EPO was pressed put them in danger of electrocution? Or force them to wait four minutes beore they can enter?

      It's not so much that the firefighters spraying water are worried about getting electrocuted via current conducting through the water itself... it's more about worrying bout stumbling i

  • The reason why some NICs don't auto-neg (Score:3, Informative)

    by phaetonic ( 621542 ) * on Thursday January 20, 2005 @03:48PM (#11423942)
    I have run across this issue in data centers numerous times. This still occurs with the latest hardware, no matter what vendor or OS. I have this problem on SunFire280Rs and Compaq DL360s. What it comes down to is the switch being used in the data center and the settings in the OS. Typically, data centers set their switch to forced 100-full (unless of course they are using fibre or Gb). The OS must be set to force its NICs in the same mode, or they will either drop alot of packets. Sounds like a disconnect in communications between the NOC and the customer.

  • OOB console access is the answer. (Score:3, Insightful)

    by Mordant ( 138460 ) on Thursday January 20, 2005 @03:52PM (#11423981)
    They ought to have out-of-band (OOB )serial-console access to their servers via a terminal server for any number of reasons, including this one; if they'd implemented OOB console access, they could've sshed into the terminal server, gotten onto the consoles of the servers in question, and used ifconfig to fix the duplex issue.

    Why they don't seem to grasp this is beyond me . . . anyone running a public-facing, high-volume service should have OOB access to all servers, routers, switches, firewalls, etc. . . . it's just common sense.
  • I told you so. [slashdot.org]

    Looks like my "Newbie Operator" found hisself a new job.

  • 2 accounts of the powerloss (Score:5, Funny)

    by Spazholio ( 314843 ) <slashdot@nOSPAM.lexal.net> on Thursday January 20, 2005 @03:56PM (#11424028) Homepage
    The one [livejournal.com] they tell you about and the real [livejournal.com] one.

  • No! (Score:3, Insightful)

    by Saeed al-Sahaf ( 665390 ) on Thursday January 20, 2005 @03:57PM (#11424031) Homepage
    embedded NICs...

    Who in their right mind goes with the on-board NIC in a server environment?

    • Re:No! (Score:3, Interesting)

      by juuri ( 7678 )
      Who in their right mind goes with the on-board NIC in a server environment?

      Are you kidding?

      How about everyone? Regardless of PC, Sun, Alpha or whatever hardware.
      • Does not mean it's a good idea! Not a single machine where I work uses the on-board NIC, from servers down to desktops. And all of our machines have a two year lifecycle, tops. We generally plug in a 3Com card of some type.

        • Re:No! (Score:2, Informative)

          by SenorChuck ( 457914 )
          On all of the (actual) servers I've worked with, the onboard NICs are exactly the same hardware that you get with the server-grade PCI NICs.
        • Does not mean it's a good idea! Not a single machine where I work uses the on-board NIC, from servers down to desktops. And all of our machines have a two year lifecycle, tops. We generally plug in a 3Com card of some type.
          The smallest of the Sun 1U rackmount Sparc servers do not even have a PCI slot to take a NIC -- no expansion at all, but two on-board 100M interfaces are plenty for most data center deployments of these small boxes.
  • I'm surprised that they didn't have their own little UPSes to bring the system down cleanly before. Sure, the facility is supposed to provide power at all times, even if there's a power grid interruption, but that doesn't get tested very often and isn't under your control. Furthermore, in the event that the facility's power is actually going to go out, there isn't any way for the machines to find this out and shut down cleanly.

    • Re:No UPSes before? (Score:3, Informative)

      by Nonesuch ( 90847 )

      I'm surprised that they didn't have their own little UPSes to bring the system down cleanly before. Sure, the facility is supposed to provide power at all times, even if there's a power grid interruption, but that doesn't get tested very often and isn't under your control. Furthermore, in the event that the facility's power is actually going to go out, there isn't any way for the machines to find this out and shut down cleanly.

      Unfortunately, this would defeat the purpose of the "Big Red Button", which

  • Accidents happen (Score:3, Interesting)

    by Migraineman ( 632203 ) on Thursday January 20, 2005 @04:07PM (#11424176)
    About a decade ago, we had a series of "incidents" with the EPO button in the software lab. Shortly after a serious lab upgrade (due to constantly blowing breakers,) someone decided to test the EPO switch (it was a bit of a novelty at the time.) *click* "Cool, it works. Hey, how do you reset this thing?" Turns out you needed to have a key to reset it. It took about 4 hours to find someone who had the key. That one got replaced with the Mark II resetable switch ...

    About a month later, one of the managers was giving a prospective new-hire a tour. He got to the software lab, and started blathering about "don't ever push the red switch" as he put his finger on the switch ... *click*

    So some einstein decided that the Big Red Switch was "dangerous" and put a plexi cover over it - the same kind that goes over the thermostat control, and the same kind that has a key lock. Yep, about six months later we had a gen-you-ine emergency. One of the HP 9000/300 monitors went crispy, and was snorting smoke and sparks. One of the software folks went to hit the Big Red Button, but was somewhat nonplussed to find a locking cover over it. She took the co-located fire bottle, sheared the cover off, pressed the button, then got to use said fire bottle on the monitor.

    So the cover gets replaced again, though this time with a non-locking cover. At some point, the software server stack needed to be relocated into the corner with the Big Red Button. Another einstein discovered that it was inconvenient to slink behind the equipment rack - the cover kept bashing him in the neck or shoulder. So he removed it, thinking that accidental presses wouldn't happen because the button was obstructed by the server stack. (yep, inaccessible = useless.) Some time later, the equipment was being jockeyed for an upgrade, and one of the big SCSI cables snagged the Big Red Button and *click* ...

    All these shenanigans happened in the space of one year, and I got tired of the thrash. I measured the space between the back of the switch and the faceplate - just over 3/4 inch. I cut a horseshoe shape out of 3/4 plywood, and hung it on the switch shaft. In and emergency, it's really easy (and obvious) to remove it. Gravity keeps it there otherwise. No problems since ...
  • Maybe people will see this and relise the LJ staff are geeks, unlike most of their fanbase, so while you maybe mocking their minions they can still bring down a server looking at a single article with the rest of us slashdotters.
  • We have one of those Big Red Buttons in our datacenter (about 7 feet up on the wall, so no one could accident bump into it). About a year after it was installed, an electrician showed up to do something in the ceiling, and accident leaned his ladder up against our exposed Big Red Button.

    Needless to say, we now have a cover over our Button. Funny thing is, the electrician who installed the original button is also the guy who leaned his ladder against it.

  • ...when you buy crappy [pcchipsusa.com] kit [ecsusa.com]. Next time do [asus.com] it [ibm.com] right [apple.com].
  • OK, this _shouldn't_ apply to a good, reputable datacenter that has structured wiring to TIA/EIA-568 running gigabit.

    I most often see autoneg problems with faulty cabling (split pairs from crimps). 98% of newbies cannot get it right, and they aren't to blame because the standards are counter-intuitive unless you've worked for Ma Bell for 40+ years. I beware of all field crimps.

    OTOH, I saw one example of a Crisco Crapalyst router not wanting to play with some devices. Of course they blamed the device, bu

  • (a) Manager that pushed the "off" button gets promoted.
    (b) Engineers that spent their weekends getting the system back up: off to India with your jobs!
  • I assume that they will have the responsible luser pay for the down time plus the 2 weeks credit plus the extra hours for the staff to bring the system up.

    And what the hell was a visitor doing playing with the Big Red Button anyways?
  • This happened to us last year in our datacenter.

    The Facilities manager had some guys in to install shelving to store toner, cables, etc.

    Our datacenter is divided into two sections, inner and outer. All CPUs, UPSs, HVAC, etc are in the inner room. The outer room is shelving, desks, CCTV (security), etc.

    The EPOs are near every door, as they should be, including the outer doors. Some guy, while installing the shelves, decided to take a little break and lean against the wall, leaning on the EPO in the proce

Slashdot Top Deals

If you steal from one author it's plagiarism; if you steal from many it's research. -- Wilson Mizner

Close