Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
The Internet Communications

Data Center Move Goes Awry for TypePad 104

Posted by CowboyNeal from the sloooow-responses dept.
miller60 writes "Problems during a switchover between data centers have slowed TypePad, the popular blog hosting service. Typepad maxed out its data center space and all available power at Internap, and is in the process of moving to a new data center. The transition has not gone smoothly, causing the Typepad service to slow to a crawl amid very public complaints by its blogger customers. TypePad operator Six Apart promises things will improve soon."
This discussion has been archived. No new comments can be posted.

Data Center Move Goes Awry for TypePad More

Data Center Move Goes Awry for TypePad

Comments Filter:

  • Well.. (Score:5, Funny)

    by Cruithne ( 658153 ) on Friday October 28, 2005 @12:02AM (#13894539)
    I'm sure our linking to them will help immensely with the slowness!

    • Re:Well.. (Score:5, Informative)

      by TubeSteak ( 669689 ) on Friday October 28, 2005 @12:43AM (#13894669) Journal
      I wasn't going to click their link.... until you said that

      Then i read the farking article and realized it didn't matter

      Typepad's home page is hosted at Akamai, a major content distribution network, and has been spared the level of performance problems seen by many TypePad bloggers. Mena Trott promised things will be better soon. /blockquote

  • It's what they do (Score:5, Funny)

    by bypedd ( 922626 ) on Friday October 28, 2005 @12:06AM (#13894552)
    "...amid very public complaints by its blogger customers"

    But that's why we love bloggers! We wouldn't expect any less.

  • Oh, real smart... (Score:5, Funny)

    by VE3MTM ( 635378 ) on Friday October 28, 2005 @12:06AM (#13894554)
    1. Hear about ailing server
    2. Post story about said ailing server on /., including a link to the site.
    3. ???
    4. Whatever you're expecting here, it ain't profit.

    • The TypePad home page is on Akamai ... (Score:5, Informative)

      by miller60 ( 554835 ) on Friday October 28, 2005 @12:26AM (#13894623) Homepage
      ... hence the link to that URL rather than direct link to individual blogs. If a midnight Slashdotting brings Akamai and its 15,000 servers to its news, I guess THAT would be news.

      Hey, if it happens, we'll have to post a link about the Akamai outage and see if it happens again.

    • hey we always try and kick them when their down.. they are blogers after all.. blogers that pay for it too..
    • The Slashdot effect might take down small personal servers on shared hosting (e.g. where there's 10,000 websites on one machine running at capacity), but it barely touches major websites. e.g. the current stats are that a Slashdot hit increases the visitor count temporarily by 18-20% on some major sites.

      Whooopie.

      It this case the problem is a lack of capacity (even though the link is directed at the problem), but I doubt Slashdot's herding compares with the continual massive herding in the blog sphere.

  • Comment removed (Score:5, Funny)

    by account_deleted ( 4530225 ) on Friday October 28, 2005 @12:10AM (#13894562)
    Comment removed based on user account deletion

    • Re:Phew. (Score:4, Funny)

      by bladesjester ( 774793 ) <.slashdot. .at. .jameshollingshead.com.> on Friday October 28, 2005 @12:29AM (#13894634) Homepage Journal
      For anyone who gets a giggle out of making fun of the emo kids (and I know most of us probably do), www.emorangers.com is truly amusing.

      They have the opening sequence to a "show" called Mighty Morphin Emo Rangers there. Just put down the soda first...

    • Re:Phew. (Score:3, Informative)

      by somethinghollow ( 530478 )
      What the hell are you talking about? 10% of myspace's functionality is down at all times due to Tom's apparent love of testing his (or whoever the developers are now) newest script on the live server instead of a development server.

      I'm not emo or anything (all my friend's use myspace, I swear that's why I joined). But I did send 4 messages to the same person today (at least 1 minute apart) and ended up at a page that said "test"... just the word "test"... no back ground or navigation... just the word "
      • Id like to $_$ in on the latest in fads but i too am short of free time :/

        mabey if i take your spare time, and add my spare time, then we could make a social network taht kicked ass... something better than Orkut

        then again why bother when the damn thing will do nothing but have portugese people sending me damn questions.
      • I'd help you write you a better one but I'm too busy subscribing to all those social network things to have a social life anyway...

  • typepad gone bad (Score:3, Informative)

    by ak_hepcat ( 468765 ) <slashdotNO@SPAMakhepcat.com> on Friday October 28, 2005 @12:15AM (#13894582) Homepage Journal
    I wondered why I wasn't able to read WWdN:iX today...

    And here Wil had been considering moving to TypePad for his next site upgrade....

  • We think. They complain about everything. (Score:4, Funny)

    by TCQuad ( 537187 ) on Friday October 28, 2005 @12:17AM (#13894588)
    ...causing the Typepad service to slow to a crawl amid very public complaints by its blogger customers.

    Or so we assume. Of course, we can't actually access the blogs to double check, but it's a safe bet.
  • Maybe they need to change their name to "StickyPad".

  • Heh, I did that. Twice (Score:5, Interesting)

    by Anonymous Coward on Friday October 28, 2005 @12:19AM (#13894597)
    In the dot bomb era, I was working at a very large porn site. We moved our site twice to escape bills. This is essentially how it goes down:

    1. Get all your DNS updates ready.
    2. Shut everything down all at once at your low peak use time.
    3. Submit your DNS changes (or move your own dns servers in advance to new location).
    4. As fast as you can pile servers, routers, firewalls, switches, etc into vans, trucks, etc.
    5. Drive at 80 miles an hour to new data center hoping to fuck that pot hole you just hit didn't fuckup some scsi drives.
    6. As fast as you can unload all servers and re-set up your system.
    7. Spend about 24-48 hours fixing shit.
    8. Get drunk.

    I'm not kidding. I did this twice with more than 100 servers.

    • Re:Heh, I did that. Twice (Score:3, Interesting)

      by Anonymous Coward
      Oh, one other thing I forgot to mention regarding this move (first one only, by the second one there was hardly anyone left). This was the one time you got to boss the owner, sales people and marketroids around all at once. Because everyone, and I mean everyone, was invited to this party.

    • Re:Heh, I did that. Twice (Score:5, Interesting)

      by LogicX ( 8327 ) * <slashdot AT logicx DOT us> on Friday October 28, 2005 @04:15AM (#13895153) Homepage Journal
      I also did some crazy datacenter moves -- they went a little different:

      Squeeze as many virtual hosts onto as few boxes as possible.
      Move the cleared off boxes to the new datacenter, get them up and running, with a ton of IPs
      Move swaths of virtualhosts and users at a time --
      update DNS to point to the new box, create www2.domain pointing to the new box
      update old box virtualhost with a 301 redirect of everything to www2.domain

      All traffic moves to the new NOC, all that remains are redirects for a few hours while dns caches around the world updating.

      Great method for moving hundreds of MB/sec of traffic in no time.
      • True, but given that the GP said "to escape bills", I suspect the initial moved boxes would tip off the colo operator that you're about to run out on them, and to stop you cold.

      • Re:Heh, I did that. Twice (Score:1, Interesting)

        by Anonymous Coward
        Well, you should've used a 302 redirect (temporary) to www2.domain.com, not a 301 (permanent). And, as another poster pointed out, you should have lowered your DNS TTL to a small value (10 minutes) well before the move (at least the value of the old TTL, plus a little extra, just to be safe). Other than that, good plan.
        • you should have lowered your DNS TTL to a small value (10 minutes) well before the move

          Or, if you were exceptionally clever, you would have used Dan Bernstein's tinydns and its time-to-die / timestamp support to just schedule when the DNS switch would happen, and it would magically take care of adjusting the TTL appropriately so the switchover occured at the second specified.

          Mmmm, tinydns....

          --
          lds

    • Happens all the time in pr0n hosting biz (Score:4, Interesting)

      by anticypher ( 48312 ) <anticypher.gmail@com> on Friday October 28, 2005 @05:32AM (#13895324) Homepage
      I just had a one-time client do this. Called me up one day this summer, asking if I could help them move their data centre that very evening. "Sure," I said, "if you pay me cash up front", and they did, not even negotiating my obvious overcharging.

      Legitimate data centres around Europe don't let anyone take out machines until everyone agrees all bills have been paid. It limits the damage from pr0n websites pulling this stunt. The courts had seized all their bank accounts and given the money over to the data centre, the ISPs, and all the rest of their creditors. They actually had quite a large stash of money, but the boss was a big time cheat who just didn't like paying bills. Once their bills had been paid, they were told they had 24 hours to clear out their operation.

      It was a disaster, of course. Their DNS $TTL was a week, they had all kinds of affiliate programs who broke for a while. The new data centre was an old office building in a dodgy office park, so it didn't yet have the cooling for 3000 servers or a redundant electricity supply. There was a single fibre connection passing nearby, and I had to find 200 metres of monomode to get fused and in operation in a matter of hours. While I set up their new data centre routers and switches, they hired a bunch of students to load up a couple of moving vans starting at midnight. Piles and piles of cheap, crappy DIY servers, and two huge cardboard boxes of cables. Then they drove 230 Kms, arriving at 7:00 AM, and started setting things back up. By noon, they had only 150 servers back up and running.

      I think they had over 50% machine failure and it took them 2 or 3 days just to install the 2500+ machines in the new area. They did lose most of their customers, but wrote it off as normal churn in the pr0n hosting biz.

      the AC

    • Re:Heh, I did that. Twice (Score:4, Insightful)

      by kju ( 327 ) on Friday October 28, 2005 @06:32AM (#13895436)
      Often this step should be included before all others:

      0. Update your DNS zones and lower the TTL to e.g. 10 Minutes. Otherwise people might not notice the new address for hours or even days (depending on your normal settings).

    • I'm a little skeptical here. Why would a porn site need 100 servers or even anything even close to that? That is insane. I could understand the need for bandwidth and storage space but not nothing more then a handful of actual servers.
      • Most porn sites are hosted by big porn-friendly (and porn-specific) hosting companies. They tend to offer package deals -- server space, all the scripts required to accept payment and handle accounts, and sometimes even porn to sell. (Ever wonder why a lot of sites have the same pictures? That's why.)

        It's because most hosting companies have some restrictions on "naughty" content in their TOS; or they did, last time I was in the market. So if your business depends on hosting porn, you really want a host
        • I was trying to compare this to what a typical large usenet providers might have. Although the delivery method is different, the usage patterns are about the same. They have massive amounts of bandwidth and storage that would probably dwarf any porn web site. I can not find any current usenet providers equipment lists but in the past, the norm was nothing more then a few Sun boxes. usenetserver.com has some actual stats online [usenetserver.com] like total connections and outbound bandwidth but they mention nothing about
  • nobody was reading your stupid blog anyways?

  • Living up to their name (Score:3, Funny)

    by Recovering Hater ( 833107 ) on Friday October 28, 2005 @12:23AM (#13894610)
    "Typepad maxed out its data center space and all available power at Internap..." Am I the only one that finds the name Internap a little ironic here?

    • Re:Living up to their name (Score:5, Informative)

      by Anonymous Crowhead ( 577505 ) on Friday October 28, 2005 @12:34AM (#13894650)
      Internap's facilities are pretty big. When we moved our stuff into their Seattle facility about three years ago, they were at 60% capacity. Now they are at 97% capacity. This facility has hundreds upon hundreds upon hundreds of racks each of which is capable of holding 48 1U servers. They recently gutted a conference room and a large staging area space so they could add more. I believe Internap has 25 similar facilities worldwide. They are doing brisk businesss.
    • Am I the only one that finds the name Internap a little ironic here?

      NAP == Network Access Point

      If you understand that InterNAP's business model is to provide premium connectivity services (they buy bandwidth on all the big backbones and use that paid-for-bandwidth to provide guaranteed connectivity and all kinds of fancy routing-tricks to their customers) it actually makes sense that leaving InterNAP's service would result in crappy network performance.

      Disclaimer: I am a shareholder of INAP, I bought shedlo

  • A Solution! (Score:5, Funny)

    by Macblaster ( 94623 ) on Friday October 28, 2005 @12:27AM (#13894626) Homepage
    If Typepad was utilizing Sun's $1 per gigabyte and per hour of processing time [slashdot.org] then such slowdowns wouldnt be a problem...

  • Do your research... (Score:5, Insightful)

    by JohnA ( 131062 ) <johnanderson&gmail,com> on Friday October 28, 2005 @12:42AM (#13894665) Homepage
    Before we committed our 3 racks of servers to a data center, one of our primary research factors was the availability of growth capacity for our data center.

    To that end, we chose to host our servers in an Emeryville data center rather than the same provider's Sacramento data center.

    They were able to contractually guarantee 5 more racks, and 100 more amps of power, giving us the room to grow.

    You can buy cheap, but it ends up biting you in the butt. Make sure that you look at your long-term goals when deciding where to set up shop.
    • My real question is if they really have that many servers, that much traffic, and are pulling that much power.. have they ever thought of hosing in more than one data center.. they could have set up a few servers at the new place and slowly move over services with out people noticing - instead they did a major swap and now getting screwed by their lack of design, forthough, i don't know.. jsut seems dumb to me..

  • You get what you pay for (Score:3, Informative)

    by davidwr ( 791652 ) on Friday October 28, 2005 @12:42AM (#13894668) Homepage Journal
    The least intrusive way to move data centers is by using remote hot backups. That costs. A lot.

    The second best option is to estimate your downtime, double it, and raise it to the next unit of measurement. If you think it will take 2 days to move if everything goes according to plan, tell your customers it will take up to 4 weeks to iron out all the kinks.
    • If you think it will take 2 days to move if everything goes according to plan, tell your customers it will take up to 4 weeks to iron out all the kinks.

      Ahh, so you're using the Scotty Manuver.

  • I hate subject lines (Score:4, Funny)

    by DanThe1Man ( 46872 ) on Friday October 28, 2005 @12:53AM (#13894693)
    How do bloggers complain when they can't blog? Are they smart enough to handle email?
  • Internap is by far one of the best hosts out there, and I believe they are expanding their facilities to keep up with demand. Instead of jumping ship, Typepad should have waited for Internap's capability to provide more space/power... If they could not wait, then they should have brought up a few boxes at temporary locations until they could move them back to Internap. Anyone with experience with Internap already knows that they are probably the best of the best...

    Kyler
  • I host my own blog using WordPress on my own domain. No problems with getting slashdotted as it is only linked from a quite obscure site.

    Come to think of it, I might was well blog to a .txt file using Notepad on my standalone laptop.

  • Six Apart?? (Score:5, Interesting)

    by Khyber ( 864651 ) <techkitsune@gmail.com> on Friday October 28, 2005 @01:53AM (#13894841) Homepage Journal
    TypePad operator Six Apart promises things will improve soon.

    I don't want to sound like a troll post. Sadly, there may be no other way to actually make sense with my question. We're talking about the same Six Apart that consistently gets in the way of free speech, suspends accounts, paid or not, for the absolute most whimsical reason, and the very same Six apart that will delete your community if it doesn't serve their consumerist policy? (like my LJ community DIERIAA {legal links to music offered freely for public download on the net by the record labels/copyright owners themselves} was terminated for no reason, never even had it's first post made, it just got wiped out, just for it's name.)

    We're talking about THAT Six Apart, right? The one that doesn't give half a shit about you, as long as they can steal your money Six Apart? Thank the deities that I never paid for one of their accounts. I've said worse on my webpages and not once has the Secret Service complained, not until Six Apart took over LiveJournal, that is.

    I apologize for this seeming like a troll post. I've had my own horrible experiences with LJ/SA, and for once, I had a chance to vent out my frustrations, and let it be known what I see LJ/SA to be. This is a personal opinion, so if you are in possession of better experiences, let them be known and put me into my place, please. But I still wish to know.. Improve? What are they going to improve upon, their volunteer idiots to ban and suspend journals or communities without even following their own set policies? Surely you're joking.

    • Re:Six Apart?? (Score:1, Interesting)

      by Anonymous Coward
      I work at 6A, and it's a great place with a lot of great people. If your bad experiences come from the LJ abuse team, they don't have their policies set by 6A so don't blame them. Sorry the abuse volunteers aren't very understanding, they're an odd bunch.
      • It still falls on SixApart's shoulders, as they now own LiveJournal.

        In all seriousness. If SixApart expects LJ to ever stop being a pink ghetto, they're going to have to start taking complaints like this seriously. LJ's "abuse department" has a long history of making screwy, inconsistent decisions that would (and should) appaul anybody at SixApart who cares about their corporate self-image and goodwill. From what I can see, this has not changed since SixApart took over.

        So, yes, it's proper to blame SixAp
    • We're talking about the same Six Apart that consistently gets in the way of free speech, suspends accounts, paid or not, for the absolute most whimsical reason, and the very same Six apart that will delete your community if it doesn't serve their consumerist policy?

      Free speech? That has nothing to do with it (at least not in the way you think it does). 6A has no means by which to stifle your rights to communicate with the world, they only have the ability to impact how you make use of the private system
    • (like my LJ community DIERIAA {legal links to music offered freely for public download on the net by the record labels/copyright owners themselves} was terminated for no reason, never even had it's first post made, it just got wiped out, just for it's name.)

      Wow, you're a conspiracy theorist. Hint: they probably wiped the group because of the offensive name (it looks like a poor attempt at spelling "diarrhea.")


  • Good thing The Speaker of the House didnt use TypePad!
  • It only scares me because we're moving our data center and 30 users. Of course we've taken the time to plan and map contingency but I've still got this creepy feeling.
  • very public complaints by its blogger customers.

    And we all know what happens when you piss off the blogosphere.

  • Maybe people should get a real webhosting account and stop bitching.

Slashdot Top Deals

Arithmetic is being able to count up to twenty without taking off your shoes. -- Mickey Mouse

Close