Data Center Move Goes Awry for TypePad 104
miller60 writes "Problems during a switchover between data centers have slowed TypePad, the popular blog hosting service. Typepad maxed out its data center space and all available power at Internap, and is in the process of moving to a new data center. The transition has not gone smoothly, causing the Typepad service to slow to a crawl amid very public complaints by its blogger customers. TypePad operator Six Apart promises things will improve soon."
Well.. (Score:5, Funny)
Re:Well.. (Score:5, Informative)
Then i read the farking article and realized it didn't matter
It's what they do (Score:5, Funny)
But that's why we love bloggers! We wouldn't expect any less.
Oh, real smart... (Score:5, Funny)
2. Post story about said ailing server on
3. ???
4. Whatever you're expecting here, it ain't profit.
The TypePad home page is on Akamai ... (Score:5, Informative)
Hey, if it happens, we'll have to post a link about the Akamai outage and see if it happens again.
Re:Oh, real smart... (Score:1)
Slashdot effect is grossly overrated (Score:2, Interesting)
Whooopie.
It this case the problem is a lack of capacity (even though the link is directed at the problem), but I doubt Slashdot's herding compares with the continual massive herding in the blog sphere.
Comment removed (Score:5, Funny)
Re:Phew. (Score:4, Funny)
They have the opening sequence to a "show" called Mighty Morphin Emo Rangers there. Just put down the soda first...
Re:Phew. (Score:2)
Re:Phew. (Score:3, Informative)
I'm not emo or anything (all my friend's use myspace, I swear that's why I joined). But I did send 4 messages to the same person today (at least 1 minute apart) and ended up at a page that said "test"... just the word "test"... no back ground or navigation... just the word "
Re:Phew. (Score:2)
mabey if i take your spare time, and add my spare time, then we could make a social network taht kicked ass... something better than Orkut
then again why bother when the damn thing will do nothing but have portugese people sending me damn questions.
Re:Phew. (Score:2)
typepad gone bad (Score:3, Informative)
And here Wil had been considering moving to TypePad for his next site upgrade....
We think. They complain about everything. (Score:4, Funny)
Or so we assume. Of course, we can't actually access the blogs to double check, but it's a safe bet.
Slow TypePad means... (Score:2)
Heh, I did that. Twice (Score:5, Interesting)
1. Get all your DNS updates ready.
2. Shut everything down all at once at your low peak use time.
3. Submit your DNS changes (or move your own dns servers in advance to new location).
4. As fast as you can pile servers, routers, firewalls, switches, etc into vans, trucks, etc.
5. Drive at 80 miles an hour to new data center hoping to fuck that pot hole you just hit didn't fuckup some scsi drives.
6. As fast as you can unload all servers and re-set up your system.
7. Spend about 24-48 hours fixing shit.
8. Get drunk.
I'm not kidding. I did this twice with more than 100 servers.
Re:Heh, I did that. Twice (Score:3, Interesting)
Re:Heh, I did that. Twice (Score:2)
Re:Heh, I did that. Twice (Score:5, Interesting)
Squeeze as many virtual hosts onto as few boxes as possible.
Move the cleared off boxes to the new datacenter, get them up and running, with a ton of IPs
Move swaths of virtualhosts and users at a time --
update DNS to point to the new box, create www2.domain pointing to the new box
update old box virtualhost with a 301 redirect of everything to www2.domain
All traffic moves to the new NOC, all that remains are redirects for a few hours while dns caches around the world updating.
Great method for moving hundreds of MB/sec of traffic in no time.
Re:Heh, I did that. Twice (Score:2)
Re:Heh, I did that. Twice (Score:2)
That's how. I'm not talking bullshit at all.
Re:Heh, I did that. Twice (Score:3, Informative)
Actually, "force" is exactly how. If you're delinquent on your bills a colo facility, the fine print in your contract generally states that the operator can prevent you from entering their private property and accessing your gear. If you can still walk into the building, they usually can change com
Re:Heh, I did that. Twice (Score:1, Interesting)
Re:Heh, I did that. Twice (Score:2, Interesting)
Or, if you were exceptionally clever, you would have used Dan Bernstein's tinydns and its time-to-die / timestamp support to just schedule when the DNS switch would happen, and it would magically take care of adjusting the TTL appropriately so the switchover occured at the second specified.
Mmmm, tinydns....
--
lds
Happens all the time in pr0n hosting biz (Score:4, Interesting)
Legitimate data centres around Europe don't let anyone take out machines until everyone agrees all bills have been paid. It limits the damage from pr0n websites pulling this stunt. The courts had seized all their bank accounts and given the money over to the data centre, the ISPs, and all the rest of their creditors. They actually had quite a large stash of money, but the boss was a big time cheat who just didn't like paying bills. Once their bills had been paid, they were told they had 24 hours to clear out their operation.
It was a disaster, of course. Their DNS $TTL was a week, they had all kinds of affiliate programs who broke for a while. The new data centre was an old office building in a dodgy office park, so it didn't yet have the cooling for 3000 servers or a redundant electricity supply. There was a single fibre connection passing nearby, and I had to find 200 metres of monomode to get fused and in operation in a matter of hours. While I set up their new data centre routers and switches, they hired a bunch of students to load up a couple of moving vans starting at midnight. Piles and piles of cheap, crappy DIY servers, and two huge cardboard boxes of cables. Then they drove 230 Kms, arriving at 7:00 AM, and started setting things back up. By noon, they had only 150 servers back up and running.
I think they had over 50% machine failure and it took them 2 or 3 days just to install the 2500+ machines in the new area. They did lose most of their customers, but wrote it off as normal churn in the pr0n hosting biz.
the AC
Re:Happens all the time in pr0n hosting biz (Score:1)
Re:Happens all the time in pr0n hosting biz (Score:2)
the AC
Re:Heh, I did that. Twice (Score:4, Insightful)
0. Update your DNS zones and lower the TTL to e.g. 10 Minutes. Otherwise people might not notice the new address for hours or even days (depending on your normal settings).
Re:Heh, I did that. Twice (Score:1)
Hosting companies don't like porn (Score:2, Informative)
It's because most hosting companies have some restrictions on "naughty" content in their TOS; or they did, last time I was in the market. So if your business depends on hosting porn, you really want a host
Re:Hosting companies don't like porn (Score:1)
Am I the only one thinking... (Score:2, Funny)
Re:Am I the only one thinking... (Score:1)
Living up to their name (Score:3, Funny)
Re:Living up to their name (Score:5, Informative)
Re:Living up to their name (Score:3, Informative)
NAP == Network Access Point
If you understand that InterNAP's business model is to provide premium connectivity services (they buy bandwidth on all the big backbones and use that paid-for-bandwidth to provide guaranteed connectivity and all kinds of fancy routing-tricks to their customers) it actually makes sense that leaving InterNAP's service would result in crappy network performance.
Disclaimer: I am a shareholder of INAP, I bought shedlo
A Solution! (Score:5, Funny)
Do your research... (Score:5, Insightful)
To that end, we chose to host our servers in an Emeryville data center rather than the same provider's Sacramento data center.
They were able to contractually guarantee 5 more racks, and 100 more amps of power, giving us the room to grow.
You can buy cheap, but it ends up biting you in the butt. Make sure that you look at your long-term goals when deciding where to set up shop.
Re:Do your research... (Score:1)
You get what you pay for (Score:3, Informative)
The second best option is to estimate your downtime, double it, and raise it to the next unit of measurement. If you think it will take 2 days to move if everything goes according to plan, tell your customers it will take up to 4 weeks to iron out all the kinks.
Re:You get what you pay for (Score:3, Funny)
Ahh, so you're using the Scotty Manuver.
I hate subject lines (Score:4, Funny)
Re:I hate subject lines (Score:2, Funny)
Re:I hate subject lines (Score:1)
They post on Slashdot.
As a blogger, I always find myself wondering how everyone else complains.
Re:I hate subject lines (Score:2)
They shouldnt have left Internap (Score:1, Interesting)
Kyler
I host my own blog. (Score:1)
Come to think of it, I might was well blog to a
Re:I host my own blog. (Score:3, Funny)
Hey, I might as well patent that. With the USPTO as it is, I might just get away with it.
Re:I host my own blog. (Score:2)
aka a dairy: a blog that no one else reads.
which means theres an lot more diaries out there than blogs i suspect...
Re:I host my own blog. (Score:5, Funny)
That's a pretty cheesy quote, son, but then again, you have to milk these things for all they're worth.
Re:I host my own blog. (Score:1)
Six Apart?? (Score:5, Interesting)
I don't want to sound like a troll post. Sadly, there may be no other way to actually make sense with my question. We're talking about the same Six Apart that consistently gets in the way of free speech, suspends accounts, paid or not, for the absolute most whimsical reason, and the very same Six apart that will delete your community if it doesn't serve their consumerist policy? (like my LJ community DIERIAA {legal links to music offered freely for public download on the net by the record labels/copyright owners themselves} was terminated for no reason, never even had it's first post made, it just got wiped out, just for it's name.)
We're talking about THAT Six Apart, right? The one that doesn't give half a shit about you, as long as they can steal your money Six Apart? Thank the deities that I never paid for one of their accounts. I've said worse on my webpages and not once has the Secret Service complained, not until Six Apart took over LiveJournal, that is.
I apologize for this seeming like a troll post. I've had my own horrible experiences with LJ/SA, and for once, I had a chance to vent out my frustrations, and let it be known what I see LJ/SA to be. This is a personal opinion, so if you are in possession of better experiences, let them be known and put me into my place, please. But I still wish to know.. Improve? What are they going to improve upon, their volunteer idiots to ban and suspend journals or communities without even following their own set policies? Surely you're joking.
Re:Six Apart?? (Score:1, Interesting)
Re:Six Apart?? (Score:2)
In all seriousness. If SixApart expects LJ to ever stop being a pink ghetto, they're going to have to start taking complaints like this seriously. LJ's "abuse department" has a long history of making screwy, inconsistent decisions that would (and should) appaul anybody at SixApart who cares about their corporate self-image and goodwill. From what I can see, this has not changed since SixApart took over.
So, yes, it's proper to blame SixAp
Re:Six Apart?? (Score:2)
Free speech? That has nothing to do with it (at least not in the way you think it does). 6A has no means by which to stifle your rights to communicate with the world, they only have the ability to impact how you make use of the private system
Re:Six Apart?? (Score:2)
Wow, you're a conspiracy theorist. Hint: they probably wiped the group because of the offensive name (it looks like a poor attempt at spelling "diarrhea.")
Re:New TypePad Plus for Those Extra-Heavy Days (Score:2)
Looks like Dennis dodged a bullet here (Score:2)
Good thing The Speaker of the House didnt use TypePad!
This scares the bejesus out of me (Score:2)
Look Ma, Everyone Links To My Complaints! (Score:2)
And we all know what happens when you piss off the blogosphere.
Re:Look Ma, Everyone Links To My Complaints! (Score:1)
Nope. Thats wrong. Usenet is web1.0 tech, and therefore cannot have cool(?) meaningless buzzwords attached to it or formed as a derivative of its name. I mean, if we did that with everything we'd have tripe like:
Oh, wait...
Re:Look Ma, Everyone Links To My Complaints! (Score:1)
Re:Look Ma, Everyone Links To My Complaints! (Score:2)
Re:Look Ma, Everyone Links To My Complaints! (Score:1)
Was the whole milk still in beta? Did the newpaper leverage you as a development asset? Is that lawn-site developed with a mashup of scalable syndicated fertilizer api's? I'm not sure you are fully web 2.0 compliant yet. I believe there is still a ghost of logical, reasoning thought floating around in there.
Here, read this site [oreillynet.com] si
Re:Look Ma, Everyone Links To My Complaints! (Score:2)
Re:Look Ma, Everyone Links To My Complaints! (Score:1)
how bout this? (Score:1)