Slashdot Back Online

I'm still not exactly clued in as to why we're back online, but hey, we are. Sometime saturday morning our Cisco router melted down. Ordinarily this would only be the end of the world, but none of our qualified personel were available to fix it, thus triggering the end of several nearby worlds as well. Props to Yazz, KurtG and Scott from Cisco for managing to help get us back online. We'll post more when we know it.

Re:Original Story

Phrases like "if you don't get this motherfucking network up within the next hour..." and "were you born clueless, or did your Mom drop you on your head when you were young..." didn't help to keep me there. I'm a contractor, not a therapist.

Jeff and Rob are not easy to work with. Best wishes to my replacement.

- Anne Tomlinson

Re:Original Story: Who are you?

I've never heard of you anne, but thats a pretty impressive troll. No doubt I'm hard to work w/. I'm a perfectionist. But neither of those phrases sounds like Jeff or I. We'd be much less polite.

Re:I went Outside!!!!

nothing is funnier when user friendly does it.

Re:Exodus

You're right, the Slashdot guys screwed up .. I want my money back, right now, you cheap Slashdot motherfuckers. How dare you run a free website without online fail-over routers? I'm goin' down to the Salvation Army and getting some free lunch and when I get back I want to see TWO rows of blinky lights in that rack, or I want a refund on my subscription, pronto.

Exodus

You are talking out of your rear.

We use Exodus, and they provide us with two separate ethernet feeds, down separate cable runs, from two separate routers in different parts of the internal NOC. No need for any routers at all; we have separate endpoint hardware on each feed and just do a rough load balance across the feeds with round robin DNS.

The recommended (by Exodus) alternative is to have a pair of peered routers which actively load balance across the feeds at the IP level, and back up each other if one goes down. I didn't do this as we're a startup I didn't want to pay for an extra pair of routers.

Either of the above will ensure that there is no single point of failure on the front end. This is referred to as a dual-homed configuration. Exodus' WAN will ensure there is no SPF further out; making your own equipment cluster and software fault tolerant is your problem :-)

It sounds like Slashdot is running with a single-homed connection, and that the router which failed is their own kit in their own rack. $$ permitting, they could have either (a) done a proper dual-homed setup, as per one of the above, or (b) had a spare router sitting in the rack and lease Exodus' managed hardware monitoring service, which would have meant Exodus techs switching it out when it failed.

I don't know what Slashdot's budget for hosting is, but we are a much smaller company than Andover and dual-homed service is not exactly killing our budget. I would conservatively assume that bandwidth is Slashdot's biggest expense.

You cannot throw pies at the co-lo provider for your own failure to have a robust setup and make proper use of the facilities they offer.

Very.

Kuro5hin is only loosely associated with OSDN. Yeah, we run their ads and get revenue that way, and we are listed in some of their material (but not all, and not often). We're not owned (even in part) by VA, and I don't know of any plans to 'buy' us (which makes sense, since they don't have to cover any costs).

I do all system administration -- DNS, mail, etc., whereas the VA owned sites all share the same pool of cool admins (like Yaz, Alliecat, etc).

Rusty and I are happy with our current colocation service (vhosting [vhosting.com]). We've never, ever had problems of connectivity (only of perl/admin error :)).
--

Original Story

the original statement was a little bit different:

I'm still not exactly clued in as to why we're back online, but hey, we are. Sometime saturday morning our Cisco router melted down. Ordinarily this would only be the end of the world, but none of our qualified personel were available to fix it, thus triggering the end of several nearby worlds as well. And when our qualified personel arrived, we discovered that she wasn't actuually as qualified as we had hoped. Then she quit, thus terminating 3 local star systems. Hemos or I will update this story as soon as we know what the hell happened. But apparently creds go to Kurt Grey and Cisco tech support. Hopefully we'll have more info soon.

Re:Story keeps changing?

Well, so far I have seen 3 different stories. And a 4th one in a comment (that looks like it fits along with the other three comments).

All the versions I've seen personally in chronological order:

I'm still not exactly clued in as to why we're back online, but hey, we are. Sometime saturday morning our Cisco router melted down. Ordinarily this would only be the end of the world, but none of our qualified personel were available to fix it. And when our qualified personel arrived, we discovered that she wasn't actuually as qualified as we had hoped. Then she quit. Hemos or I will update this story as soon as we know what the hell happened

I'm still not exactly clued in as to why we're back online, but hey, we are. Sometime saturday morning our Cisco router melted down. Ordinarily this would only be the end of the world, but none of our qualified personel were available to fix it. Hemos or I will update this story as soon as we know what the hell happened.

I'm still not exactly clued in as to why we're back online, but hey, we are. Sometime saturday morning our Cisco router melted down. Ordinarily this would only be the end of the world, but none of our qualified personel were available to fix it, thus triggering the end of several nearby worlds as well. Props to Yazz, KurtG and Scott from Cisco for managing to help get us back online. We'll post more when we know it.

The one from another comment:

I'm still not exactly clued in as to why we're back online, but hey, we are. Sometime saturday morning our Cisco router melted down. Ordinarily this would only be the end of the world, but none of our qualified personel were available to fix it, thus triggering the end of several nearby worlds as well. And when our qualified personel arrived, we discovered that she wasn't actuually as qualified as we had hoped. Then she quit, thus terminating 3 local star systems. Hemos or I will update this story as soon as we know what the hell happened. But apparently creds go to Kurt Grey and Cisco tech support. Hopefully we'll have more info soon.

/Mattias Wadenstein

Re:your cisco?

More from Roblimo over at NewsForge [newsforge.com] Sunday June 24, @07:30PM
- by Robin "Roblimo" Miller - On Saturday, June 23, the primary controller in the router that controls access to all OSDN servers hosted at the Exodus facility in Waltham, MA, suffered a catastrophic failure. The sites affected were Slashdot, freshmeat, NewsForge, and Mediabuilder, among others. The secondary controller did not automatically take over as it shoud have. It did not work when activated manually, either. The first Cisco support people contacted professed to be "amazed" at the situation, saying it was the first time they had seen a failure of this kind. OSDN and Cisco people, working through Saturday night, were unable to cure the problem. Sunday afternoon, OSDN employee Kurt Gray and Cisco rep Scott, working by telephone, were stepping through the router's configuration and, says Kurt, as they worked to undo other changes that had been made, "on one reset everything came back." OSDN network operations were already in the process of rebuilding the company's network to eliminate the router as a potential single point of failure. As of 7 p.m. US EDT most of the sites were available at least part of the time, but full service was not yet restored. There may still be slowdowns or intermttent failures until a permanent fix is made. We'll have a more complete story within a few days. Right now, OSDN network operations staff members are too busy working to talk.

Interesting (NOT!)

I wonder why /., the champions of open source, aren't using a Linux box with a few NICs and some hacked-together code as a load balancer?

IIRC, they are using a Linux box for their load ballancer. It was their router that got fried, which is a completely different beast. Heavy duty routers remain specialized boxes, and Linux hasn't really serious inroads into that market yet.

Funny that when Microsoft's router failed (probably a Cisco also) it was catastrophic incompetence but for you guy's it's just bad luck.

Not funny at all when you get the facts straight. The serious problems that MS had were with their DNS servers- which were running Windows- not their routers. IIRC the DNS servers were later cracked, too, which was rightly seen as an indication of poor security. When Microsoft uses its own products, they don't stand up to the use they're being put to, and then Microsoft has to use *BSD based systems to get working again, that's very different from when a Linux site has its non-Linux hardware melt down (and the description did make it sound like a hardware, not software, problem).

I called my ISP!

Me : Dude it's like Yahoo, it's never down.
ISP : Sa-lash dot?
Me : Dude slashdot.org!
ISP : www.
Me : No no no... listen 64.28.67.150
ISP : Uh... www
Me : Damnit I'm down can't you see I'm down?
ISP : We're like up and stuff. Is this a Macintosh?
Me : I am calling my lawyer! I'll sue you blind!
ISP : Uh I have to get my supervisor.
Me : -click-
---

Re:Original Story: Who are you?

"I'm a perfectionist"

As a perfectionist, you'll of course want to know that you use "me" when the reference comes after the verb. "Neither of those phrases sounds like Jeff or me".

Also, given that you're a perfectionist, you'll be appalled to hear that someone has been using the "CmdrTaco" identity to post poorly spelled, ungrammatical crap all over Slashdot for the last three years. That same person has always tried to justify himself by whining that he "doesn't care about that stuff" and "doesn't want to be too fussy". This may or may not be the person who wrote the notoriously buggy first release of Slash, and said that it was "close enough".

But of course that couldn't be you

Because you're a perfectionist.

Ha, ha, ha. I think I'm going to have to give up satire.

*ahem*

if that was a post about a guy, and he was thought to be less than qualified, would you be posting this?

sexism goes both ways, assuming someone isn't incompetent due to their gender is just as stupid as assuming they are

equal rights for incompetent people dammit! *L*

and yes, I am a chick

on a personal note, maybe that post was taken down due to it's rudeness, rather than the sex of the person involved...

my 2 cents...

Re:What really happened

I respectfully disagree. Give Hemos his dignity man. I conjecture that what happened was more like this...

CiscoChick: Hi Rob. It's that time again. I came by to check on your equipment.
Rob: Equipment!?! Okay. Just give me a minute to get my pants off.
CiscoChick: No, no! I meant your Cisco router. I'm here for a scheduled routine preventative maintainence checkup.
Rob: Oh! That equipment.
CiscoChick: Yeah, the router. But when I'm finished, I could check out any other hard ware you have around. <wink>
Rob: Okay. Just let me know when youre ready.

later.....

CiscoChick: Okay, Rob. I'm done checking the Cisco router.
Rob: Okay. Cool.
CiscoChick: Wow! Look at that equipment!
Rob: Yeah.
CiscoChick: I mean, it's so small!
Rob: Yeah, it's the latest new thing in miniaturization.
CiscoChick: Okay, well.... Let's not focus on the size. What is the uptime on that thing? Does it go down very often, like Windows?
Rob: Ummm... Have you ever done it in a co-location cage?
CiscoChick: No, but there's a first time for everything.


3 minutes later...

Rob: Ahhhhh! I needed that.
CiscoChick: Oh, no! What's happening!
Rob: Eeeeeeiiiiiiieeeee!!!! The router is melting!

--
"Linux is a cancer" -- Steve Ballmer, CEO Microsoft.

Hmmm.

Slipping standards in the open-source world.. ;) Didn't Microsoft catch a lot of flak for having a single point of failure that other time that most of its sites went down?

When I couldn't get my Slashdot, I assumed the worse. High-profile hijacking. Aliens beaming up the OSDN headquarters. Servers sneakily migrated to Windows, which then promptly crashed.

Kidding aside, I'm glad Slashdot is back up.

I went Outside!!!!

I was just leaving my house to go demand my News for Nerds and/or Stuff that Matters, when something hit me: THIS IS SPECTACULAR RESOLUTION!!!

When I stepped outside it looked like everything was being generated by 500,000,000 GeForce3s!!! The trees looked REAL!! It must have been at least 1,600,000,000 x 1,240,000,000!!! I couln't even see any jaggies! Talk about anti-ailiasing!!

After spending 2 days outside sue to lack of Slashdot it's hard to come back to my Power Mac 6100/60 with a 14" monitor at 640x480. I wish I had reality's 3D card...

And it seem's Slashdot has slashdotted itself. How did that happen??

--Volrath50

Re:Original Story: Who are you?

Not anouther fake identity troll!

I mean who is this CmdrTaco guy and how the heck did he get UID #1???

It's obvious that this isn't the real Rob Malda, we all know that Slashdot editors NEVER post at Slashdot....

--Volrath50

Re:Interesting

I think a better comment to make would be on the point of connection redundancy over the internet - wasn't the original military plan for the net to make sure information could still get around even following a nuclear strike......one fried router and suddenly a whole shedload of sites go down

That would be a pretty dumb comment to make in this context since the router that went down was on the premises of the customer whose sites went down.

I mean, if you step on the modem in your house, you lose your link to the internet, but that doesn't mean you've identified an Achilles' heel in the internet's infrastructure - "Aha! This single modem controls access to the ENTIRE INTERNET! By stepping on it I have rendered the whole network inaccessible to EVERYONE in my house!!"

Seems that should be pretty obvious.

Translation

Sometime saturday morning our Cisco router melted down. We were making smores, using the excess heat generated by the improperly ventilated Cisco router, Saturday morning after finishing a 16 hour coding spree when all of a sudden CmdrTaco dropped his. The ohhh so gooey marshmello and yummy chocolate dripped into the air vents and all of a sudden the Cisco smoked and all the led's went out. Realizing there must be a fault, we promptly panacked and ran around the room screeming. but none of our qualified personel were available to fix it Our Cisco expert wasn't up to the task due to eating too many smores on a earlier coding break. Upon realizing this, we attempted to fix it ourselves by plugging it back in, but to no avail. We then called Cisco. Props to Yazz, KurtG and Scott from Cisco for managing to help get us back online. Those great guys from Cisco, upon hearing of our predicament, instructed us to apply a half sinewave duration, several hundered newton, force laterally on the Cisco box using our lower right appenditure. Upon asking them "What the fuck are you talking about?" they replied "Kick the damn thing." Using extreme precision, we followed out their instructions and kicked the shit out of the box. Amazingly the thing reset itself and began working as usual. We suspect that the cause was a software bug, or a chocolate induced short circuit.

Congratulations...

It only took you forty five minutes mins to offically retract the whole "she" thing.

Gathering data from your May 2nd demographic evauation, Im thinking that you nearly lost five percent of your readers in the space of forty five minutes.

Yours in disappointment,