Forgot your password?
typodupeerror

LiveJournal Servers Go Down 596

Posted by michael
from the only-mostly-dead dept.
Wind writes "According to any journal hosted off of LiveJournal.com, the LiveJournal data center Internap has suffered a critical power failure, leaving all of LiveJournal and its content temporarily offline and requiring the revival of 100+ servers. Perhaps Six Apart wasn't quite prepared for the responsibilities of a website of this size? Updated information is posted here."
This discussion has been archived. No new comments can be posted.

LiveJournal Servers Go Down

Comments Filter:
  • Lights out (Score:5, Funny)

    by r_glen (679664) on Friday January 14, 2005 @11:31PM (#11370812)
    Sounds like someone was taking a nap over at Internap
  • by webfiend (112579) on Friday January 14, 2005 @11:31PM (#11370813) Homepage
    You can't imagine the withdrawals I'm going through. It's like the great Slashdot brownouts of '98.

    I need my fix, man!
  • Elsewhere (Score:4, Funny)

    by jm92956n (758515) on Friday January 14, 2005 @11:32PM (#11370818) Journal
    In related news, 6,000 teen-age girls were heard yelling "OMG! WTF! How will John know I life him if I can't blog about it!"
  • by Anonymous Coward on Friday January 14, 2005 @11:32PM (#11370825)
    ...the collective IQ of the internet has raised about 20 points.
  • but that's ONE HELL of a Slashdotting! :)
  • by Anonymous Coward on Friday January 14, 2005 @11:33PM (#11370827)
    and search.pl is constantly being trashed by distributed xanga botnets. perhaps michael wasn't quite prepared to be an editor of slashdot?
    • by stupidfoo (836212) on Friday January 14, 2005 @11:41PM (#11370896)
      How is this a troll? It's funny that an "editor" at site with as many problems as slashdot has feels that it isn't amazingly hypocritical to mock another site that is currently having problems. People in glass houses indeed.

      Slashdot has semi-major problems almost every day. 503 errors, "nothing for you to see here" annoyances, and a search engine that goes down more than a Thai hooker.
      • Slashdot has semi-major problems almost every day. 503 errors, "nothing for you to see here" annoyances, and a search engine that goes down more than a Thai hooker.

        Oh man, that would be one *fantastic* search engine! Is there a Google beta for this?

      • by elemental23 (322479) on Saturday January 15, 2005 @12:06PM (#11373213) Homepage Journal
        Perhaps you're new here, but italicized text in Slashdot stories is written by the story submitter. Editorial comments, if any, are not in italics. In other words, Michael didn't say anything at all in this story.

        That said, the story submitter is clearly trolling himself, as neither 6A's nor LJ's staff had anything to do with the massive power failure at their co-lo.
  • Internap is *down*? (Score:5, Informative)

    by MightyTribble (126109) on Friday January 14, 2005 @11:33PM (#11370833)
    Internap *down*?
    Bush just appointed Internap's CEO to his National Infrastructure Advisory Council [tmcnet.com], yet the man can't keep a co-lo facility switched on.

    I'm not sure what that says of Bush or of Interap. And it certainly doesn't seem to have anything to do with SixApart.

    • by slashrogue (775436) on Saturday January 15, 2005 @12:19AM (#11371125)
      Sorry, but one man can't control power to an entire co-location. I used to work for a local telecom and one day our fiber went down for about 1/3 of our customers. The reason? Some guy shootin' squirrels blew the fiber lines apart. It was on a Sunday too, when there was minimal phone tech support, I think one guy ended up fielding 350+ calls by himself.
      • by Cramer (69040) on Saturday January 15, 2005 @01:19AM (#11371448) Homepage
        I recall a story from a CP&L (now Prgress Energy) lineman about how someone had shot the fiber line strung on the high voltage tower -- between the million volt lines. As he put it, you put a pencil through the hole and line up where the "hick" (his word) proped up the rifle to shoot what he must've thought was a power line. It no fun working up there. And even less fun having to make fusion plices surrounded by million volt lines. (linemen work up there, not fiber techs.)
  • What a cock (Score:5, Insightful)

    by realdpk (116490) on Friday January 14, 2005 @11:34PM (#11370837) Homepage Journal
    "Perhaps Six Apart wasn't quite prepared for the responsibilities of a website of this size?"

    Perhaps shit happens, and a blog service doesn't warrant the necessary investment to survive whatever caused this outage?
    • well they did(they thought they had prepared for this and bought reassurance for that).
      but internap didn't deliver.

      read the site.. they were on redundant power.. which turned out to not be that redundant after all(only possible explanation really is a major cockup by someone...).
    • Re:What a cock (Score:2, Insightful)

      by qcubed (655212)
      i'm sorry, how exactly does this reflect poorly on sixapart?

      THIS doesn't reflect poorly on them. their licensing scheme for movabletype does.
    • Internap Sucks (Score:4, Interesting)

      by Nurgled (63197) on Saturday January 15, 2005 @08:49AM (#11372570)

      I seem to remember that a few years back they had a similar problem (Internap lost all power) and it turned out that some idiot had hit the big red "shut down all power to the entire datacenter" emergency button. This isn't the first time this has happened, and last time it wasn't under Six Apart's management.

      I'd say it's Internap's incompetence that caused this problem. If they can't keep their datacenter running even though they have multiple redundant power supplies then something is very wrong. I see from the outage page that LJ people are now planning to buy their own UPS so that they don't have to trust Internap anymore.

      For power outages, my house has a better record than Internap right now, and I don't even own a UPS!

      • Re:Internap Sucks (Score:3, Informative)

        by Scott Laird (2043)
        A couple points. First, there's *nothing* that you can do about the "idiot hit the big red button" problem--you're required by law to have the button, because it's a safety issue. It has to be accessible--you can't lock it in a closet. And everyone knows that if you put a big red button on a wall, sooner or later someone's going to hit it.

        I don't know what happened this time, but the ~2002 Internap Seattle outage was caused by an idiot Speakeasy tech who couldn't figure out how to use the exit door, so
  • so it's deadjournal now ?
  • by freakybob (715183) on Friday January 14, 2005 @11:34PM (#11370847)
    Well now the millions (?) of users might actually have something to write about when the servers are back up. "Today I went outside. My pupils have never been tinier..."
  • by Anonymous Coward on Friday January 14, 2005 @11:35PM (#11370848)
    Perhaps Six Apart wasn't quite prepared for the responsibilities of a website of this size?

    Ok, I understand that you don't like Six Apart; I'm no fan of their new licensing scheme either. However, I really doubt that SixApart has any control over any power failures that might occur at Internap.
  • ANGST!!! (Score:2, Funny)

    by Anonymous Coward
    Where will I write about my depression over this event?

    Oh. Slashdot.
  • That's what you get when you hire Tim Allen as your electrician at a Data Center.

    Al Borland was nailing Heidi behind the stage when the outage occured.

    Where were the APC backups?
  • by La Camiseta (59684) <me@nathanclayton.com> on Friday January 14, 2005 @11:39PM (#11370883) Homepage Journal
    Use the Coralized link [nyud.net]. No sense in crashing their status page. Plust it'll respond a lot quicker than loading the actual web page.
  • by philkerr (180450) on Friday January 14, 2005 @11:40PM (#11370893) Homepage
    I feel a great disturbance in the force..... It's as if a million bloggers cried out all at once..... and became silent.
  • by TCM (130219)
    are servers with LOM (lights out management) superior in this case?

  • by CypherXero (798440)
    It's not like most LiveJournal user's have enough to worry about, here's something for most LJ users to get melodramatic about. I'm serious, randomly pick 5 LiveJournal blogs, and I guarantee 4 out of 5 are going to be "Fuck the World" posts.
  • Mood: anxious...
  • LiveJournal's offsite status page is status.livejournal.org [livejournal.org].
  • poor internap (Score:3, Interesting)

    by Indy1 (99447) <spamtrap@fuckedregime.com> on Friday January 14, 2005 @11:53PM (#11370970) Homepage
    sounds like all the fucking spammers they host overtaxed spammer-nap's power resources and brought it all down.

    Seriously though, spammer-nap is a massive spam haus, see for yourself [spamhaus.org]

  • by ebooher (187230) on Friday January 14, 2005 @11:53PM (#11370971) Homepage Journal

    I know nothing of how InterNap is set up. I just want to throw that out there ahead of time. Now, it's time for my patent pending "Bull Shit Theory of the Day."

    Ok, here is the rant. I used to work for a Colocation facility. Nothing special, small by Telco terms. The whole facility only had about 1500 cabinets. (Though I hear they are now full, and going to be expanding.)

    We had a main power draw off of the local grid. We had a backup power draw off of the *next* cities power grid. (ie, when all the offices around us went dark, we still had power.) And you don't even want to know the kind of red tape we had to go through for *that* pull. I'm still not sure how they did it. We had fly wheel kinetic electricity storage systems, battery backups, and a diesel engine from a train so large it had it's own building.

    We used to joke that if we lost power, we had more important things to worry about. And again, we were small time compared to some of the massiveness that is out there. *cough*AADS Chicago*cough*

    So I'm kind of in agreement with the statement currently on LiveJournal. It's unknown to me how any self respecting colo facility can say "We've had a power outage that also took our redundant systems."

    I have to call bullshit on that entire train of thought. If that's true then they don't *have* any redundant systems, and I'd be looking for a new provider. The most likely thing (at least in my mind) is that someone, somewhere got mad at something specific and decided to make a point by popping the main breaker to their portion of the facility.

    Oh, that was another thing, each room had several "main" breakers. It took a hell of a power surge to pop all of them, and the Liebert systems had power filters of some kind, really really big capacitors or something I think, so a surge really never made it to the other side anyway, it got stored in the cap and then trickled out like the rest of the power.

    But I was a UNIX admin, not the EE that was planning the power generation aspects of the facility. So take some of it with grains of what ever white powdered spice you prefer.

    • Happens some days. A key breaker at a data center some of my friends work at went bad and took the whole floor with it. Generators didn't even get a shot at takeing the load because the breakage happend later in the circut. No matter how big and bad your infrastructure some points in the design will not be 100% resistent to all problems. We do our best, make plans, design good systems but the world teaches us Shit happens.
    • by CrankyFool (680025) on Saturday January 15, 2005 @12:07AM (#11371058)
      Back in the great days of the .com boom, people were building their colo facilities to insane (in a beautiful way) standards. I remember touring Exodus and Above.net (I don't know who you're referring to, though I only ever heard of above.net adopting flywheels) and being just very amused at the cool stuff they were putting in place.

      I recently (~8 months back) did some contract work for a small company whose servers were based in some colo facility in San Francisco. One of the first things I noticed was a damn heavy UPS at the bottom of their rack. Weird, I thought -- why not rely on the colo's battery system?

      Because they don't have one.

      Mind you, this was also the colo that had a cardkey system that had long ago stopped being usable, so when you needed access you used a Radio Shack $29.99 wireless intercom system and someone would come to open the door, and when you checked in they carefully wrote your name on a little nametag.

      I think standards have slipped, significantly. In some respects, this is likely a good thing -- it means you have more options now, because you can choose either the super duper "we hook up to two countries' power grids, have eight flywheels and a direct feed from microwaves in orbit" or the "err, here's your cabinet. We'll give you decent power until we don't" options.

      So ... how much are all these people paying LiveJournal again? Couldn't they request some sort of partial refund of their monthly fee?

      Oh, wait...

    • by Anonymous Coward on Saturday January 15, 2005 @12:24AM (#11371155)
      My friend's company is hosted by internap. Today he messaged me when the power went down. It was only power to the second floor, my friend's servers, while cut off to the internet were still running (on the 3rd floor). Internap has redundancy and backup generators (and enough fuel onsite to run for 30 days without external power). Apparently there was construction occuring on the second floor... my guess is that some dipshit contractor cut through a power cable or 3 and took the whole floor down.

      To all the people accusing LJ of being stupid for not having UPS systems, Internap has 3 fully redundant power systems (yes, I know, didn't help much) so most people probably don't feel the need to run their own ups.
  • by Rie Beam (632299) on Friday January 14, 2005 @11:53PM (#11370977) Journal
    Update from the site:

    "Update #1, 7:35 pm PST: we're up on 'dirty' power for now (it works, but it's unreliable)".

    Congrats to LiveJournal for assembly a coal generator in a record time.
  • Update (Score:3, Interesting)

    by TrevorB (57780) on Friday January 14, 2005 @11:54PM (#11370982) Homepage
    On the Livejournal main page:

    Update #1, 7:35 pm PST: we're up on 'dirty' power for now (it works, but it's unreliable), and we're working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We'll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We'll be working all weekend on this if we have to.

    Lovely. I just bought another year's subscription for my wife, figuring the change to Six Apart wouldn't change anything for a few months at least. LJ could lose a lot of subscribers with an outage just after the takeover.
  • by Cyburbia (695748) on Friday January 14, 2005 @11:55PM (#11370985) Homepage
    live journal is dark like my soul like my heart a void its link is cut just like i'll be doing to my arm i blame my parents
  • by MattW (97290) <matt@ender.com> on Saturday January 15, 2005 @12:02AM (#11371028) Homepage
    Er, they just announced Six Apart was buying them like days ago. I doubt they transitioned the servers in the first week.
  • by PornMaster (749461) on Saturday January 15, 2005 @12:14AM (#11371098) Homepage
    LiveJournal Servers Go Down

    With thousands of teenage girls unable to ponder in an open forum whether or not to blow their boyfriends, thousands of teenage girls go down.
  • by gmhowell (26755) <gmhowell@gmail.com> on Saturday January 15, 2005 @12:14AM (#11371102) Homepage Journal
    Because michael needs a beating. The site that rolls beta (alpha?) code onto live servers complaining and making jokes because another site goes down through no fault of its own?
    • by ces (119879)
      Really, talk about the pot calling the kettle black.

      If the datacenter that hosts Slashdot was to have a massive power failure how long would /. be down for?

      That said my company has gear in the same datacenter as LJ, our servers were back up 10 minutes after power was restored. Then again we use Oracle on HP-UX with nice SAN RAID boxes for storage for our database. So our stuff tends to recover from a sudden power loss a little better than a MySQL derivative running on clone hardware.
  • No... (Score:5, Insightful)

    by EdMcMan (70171) <moo.slashdot2.z.edmcman@xoxy.net> on Saturday January 15, 2005 @12:19AM (#11371126) Homepage Journal
    Perhaps Six Apart wasn't quite prepared for the responsibilities of a website of this size?

    What does Six Apart have to do with Internap? Livejournal has been using - and wanting to switch from - Internap for a long time.
  • by DemonWeeping (849974) on Saturday January 15, 2005 @12:49AM (#11371294) Homepage
    For those who don't know what's so hot about it and for those who think Livejournal is just a bunch of teenage girls whining.... Livejournal has just about four years of my life documented. The ease of use and the ability to "vent" is comforting, but the real value comes in the interaction. My friends see my life at their convenience and I see theirs at mine. We can choose to ignore the whining of others or we can choose to relate and comment on our own experience. Think of it this way: Open-source philosophy, emotion, and life. I put my own out there and others add to it. I add mine to others. Granted ... those quiz/meme things HAVE TO GO. I do not want to read about "what frog best resembles me" or "which 80's hair band song is me." Grrr.
  • by Bloodlent (797259) <iron_chef_sanji AT yahoo DOT com> on Saturday January 15, 2005 @12:50AM (#11371300)
    Just remember it's not ALL obnoxious, over-emotional teen-angst teenage girls. I use mine to showcase (non-depressing)poetry and make intelligent comments about intelligent topics. Basically, if someone makes an LJ about their own life, it sucks. If you can manage to write an LJ and make it about things that matter to more people than just you(ie, "Why Bush's Iraqi war is unjust" vs. "Why this babe I know should bang me"), and at the same time make it funny and enjoyable to read, then you have a good LJ. Most LJs DO suck, but there are some diamonds in the rough.
  • by wersh (765553) on Saturday January 15, 2005 @12:59AM (#11371346)

    From the article write-up (and reflecting the thoughts of quite a few of the comments I just read):

    Perhaps Six Apart wasn't quite prepared for the responsibilities of a website of this size?

    I'd love to know what makes you think this has anything to do with Six Apart. The very first line at http://www.livejournal.com states:

    Our data center (Internap, the same one we've been at for many years)...

    They've been with Internap for years, predating Six Apart's takeover. Unless LJ staff is lying, the fault here sounds like it lies entirely with Internap.

    And as far as I can tell, Six Apart didn't ditch the LJ team when they bought them out, so you probably have the exact same people working on bringing the site back up now as you would have if Six Apart had never got involved.

  • bigger explination (Score:5, Insightful)

    by moosesocks (264553) on Saturday January 15, 2005 @01:45AM (#11371537) Homepage
    I'm surprised to see that Internap's main servers are back up. It's pretty irresponsible to bring up your corporate servers before those of your clients.

    That being said, LJ's servers are back up now, but they're making sure that the databases are all in sync -- LiveJournal has one of the most massive distributed MySQL clusters in existance along with a complete caching system.

    They need to make sure that the database is all synchronized before bringing it back up -- chances are they're going to rebuild the cache too. If they didn't, the initial strain on the DB servers would probably bring the site down again.

    This does however, bring up some questions about LiveJournal's network infrastructure. Danga (the creaters of LJ, recently purchased by Six Apart) are heavy users of Perl and MySQL. Needless to say, they have made numerous contributions to both projects and have developed an innovative memory caching system for linux.

    The questions raised however, come from Perl and MySQL. Both are questionable in terms of scalability. Although I'm not qualified to comment on this, I belive that the general concensus is that MySQL is one of the least efficent databases today. Livejournal has 100+ servers. I honestly don't think that a system the size of LiveJournal should require a server cluster that big. It seems that they are trying to solve their performance/reliability problems by blindly throwing hardware at it.

    Of course, I love livejournal. It's simple, easy to use, and is a great tool for building communities. Just as it is simple, it can also be incredibly nerdy (there's actually a command prompt!). They're also completely open source.

    Hopefully, Six Apart can make their network infrastructure more 'professional' while still maintianing the community spirit that has made it so successful.
    • by Kyrrin (35570) on Saturday January 15, 2005 @05:41AM (#11372173) Homepage
      As we've said a bunch of times in the past, moving away from MySQL would be prohibitive. By now we know how to make it work for us; switching away from MySQL would not only involve massive rewriting of stuff and alterations on the existing DB, it'd take the next five years before we got as comfortable with the flaws and advantages of another DB package.

      Sure, MySQL has its flaws -- some of them pretty big -- but we can work around them.

      As for the "not needing a server cluster that big" -- do you have any clue how much data we push in an average day? We maintain so many DB clusters to improve reliability, and we maintain so many web nodes because we push a screaming shitload of traffic.
  • by Sparks23 (412116) * on Saturday January 15, 2005 @02:13AM (#11371637)
    There were already lots of LiveJournal users who were upset and confused and unhappy with the idea that LJ and Danga (the company which made LJ) had been bought by SixApart. No doubt, as there have been no downtimes of this magnitude at LJ before, doomsayers will be claiming that it's SixApart's fault.

    Never mind common sense; it won't matter that if SixApart can be held responsible for failures at InterNAP's colocation facilities, they're a much bigger -- and more powerful -- company than most people have ever given them credit for...
  • Makes me wanna laugh (Score:3, Interesting)

    by Jesus IS the Devil (317662) on Saturday January 15, 2005 @07:44AM (#11372427)
    It's funny how I was just met with some Internap sales people a few months ago. They were bragging about how their network infrastructure was superior to most others, since it intelligently routes traffic to the path of shortest response (not hops).

    They even bragged to me how their network uptime SLA is 100%! I mean good god, now I find out this is the SECOND time it's happened (from the livejournal update site)???

    I'm glad I didn't go with them...
  • by 21mhz (443080) on Saturday January 15, 2005 @07:45AM (#11372431) Journal
    The comments seem to be full of contempt for teenage -angst inane ramblings that are common on LJ. Come on. It's not like you are forced to read through this stuff.
    I have a few "friends" there at LJ, some of them net.celebs, and I like their posts. It's the matter of whose writings do you find interesting, and you are free to be completely unaware of the rest. Why all the vitriol?

The number of arguments is unimportant unless some of them are correct. -- Ralph Hartley

Working...