Beltway Prophet's Journal: Dell Power Edge 2950 Plays April Fools on ThinkGeek

Setting: Jacob finally gets to sleep at 5:15 this morning after 22 hours of uptime leading up to April Fools' Day at ThinkGeek HQ. Two hours later, his company phone starts making its "alert" sound.

7:15am

PHONE: Bleem!

SUPEREGO: Uh-oh! Somewhere, a server's in trouble!

JACOB opens eyes, looks at phone

PHONE: Bleem!

ID: I don't hear anything.

ID sings tonelessly: lalalala

PHONE: Bleem!

JACOB picks up phone, reads SMS messages

ID: Dude, Jennifer's on call. And she probably had one more hour of sleep than you. Let her deal with it. Anyway, it's just db4. I'm sure the site's fine.

PHONE: Bleem!

SUPEREGO: Yeah, but it's my responsibility too, and she might not wake up.

EGO: Well, why don't we check the site and see?

JACOB opens ThinkGeek in the phone's browser.

SUPEREGO: It seems really slow.

ID: No way, man. It's just the phone. The phone's slow!

SUPEREGO: That's true. Well, to be specific, the wi-fi reception tends to be a bit spotty. Not as spotty as my PowerBook talking to the Time Capsule at work, but...

PHONE interrupts, closes browser, opens SMS app emphatically: Bleem!

EGO: Okay, well, since this fscking thing isn't going to stop "bleeeem"ing anyway, let's just get up and deal with it.

PHONE [relieved]: Bleem!

JACOB logs on

JENNIFER [via IM]: Please tell me you are doing maintenance and forgot to shut off Nagios again.

JACOB: I wish.

JACOB wakes up a very understanding sysadmin in California, where it's three hours earlier

ENTER CHRIS

CHRIS notes the cause of the OOPS and reboots the box.

JACOB restores database replication.

Moral of the story

Always reboot your servers before a big day: uptime shmuptime. All it takes is one flaky kernel module to desynchronize your databases and ruin your morning.