Journal Journal: Dell Power Edge 2950 Plays April Fools on ThinkGeek
Setting: Jacob finally gets to sleep at 5:15 this morning after 22 hours of uptime leading up to April Fools' Day at ThinkGeek HQ. Two hours later, his company phone starts making its "alert" sound.
7:15am
PHONE: Bleem!
SUPEREGO: Uh-oh! Somewhere, a server's in trouble!
JACOB opens eyes, looks at phone
PHONE: Bleem!
ID: I don't hear anything.
ID sings tonelessly: lalalala
PHONE: Bleem!
JACOB picks up phone, reads SMS messages
ID: Dude, Jennifer's on call. And she probably had one more hour of sleep than you. Let her deal with it. Anyway, it's just db4. I'm sure the site's fine.
PHONE: Bleem!
SUPEREGO: Yeah, but it's my responsibility too, and she might not wake up.
EGO: Well, why don't we check the site and see?
JACOB opens ThinkGeek in the phone's browser.
SUPEREGO: It seems really slow.
ID: No way, man. It's just the phone. The phone's slow!
SUPEREGO: That's true. Well, to be specific, the wi-fi reception tends to be a bit spotty. Not as spotty as my PowerBook talking to the Time Capsule at work, but...
PHONE interrupts, closes browser, opens SMS app emphatically: Bleem!
EGO: Okay, well, since this fscking thing isn't going to stop "bleeeem"ing anyway, let's just get up and deal with it.
PHONE [relieved]: Bleem!
JACOB logs on
JENNIFER [via IM]: Please tell me you are doing maintenance and forgot to shut off Nagios again.
JACOB: I wish.
JACOB wakes up a very understanding sysadmin in California, where it's three hours earlier
ENTER CHRIS
CHRIS notes the cause of the OOPS and reboots the box.
JACOB restores database replication.
Moral of the story
Always reboot your servers before a big day: uptime shmuptime. All it takes is one flaky kernel module to desynchronize your databases and ruin your morning.