FortKnox's Journal: Ah Hell.... [updated]

My code was pushed to production this morning. Found an error... an update had a wrong where clause.

Yup, that's right... I completely hosed a production db this morning. Which means getting hundreds of people pissed at me to fix all the thousands of rows in the db... or getting hundreds of people pissed at me for having to do all the work the did this morning this afternoon.

Its gonna be a baaaaad day... :-(

Update: It gets better! My app is in a group of 12 (enterprise sized) applications. The DBAs set it up so recovering for a backup means they recover data for ALL 12 APPS! So I won't have hundreds of people mad at me... I'LL HAVE THOUSANDS!

Boss talked to DBAs... gave'm time to sort it out so that only my one table is recovered and nothing else.

Reminds me

[blinder]

of my favorite quote from Ron Howard's "The Paper"

"Sometimes you can just smell a horrendously shitty day."

I've been there, and have done that before... to the point where an entire code base was lost, and so I got to spend the entire night and the following day recreating all that was lost. Toughest 28 hour work day I ever put in :)

Dude

[stoolpigeon]

I ran an update on a production database once with no where clause at all. That was a fine morning. Nothing like that unexpected opportunity to test out my ability to recover from backup.

Re:

[account_deleted]

Comment removed based on user account deletion

Re:Dude

[Tet]

I ran an update on a production database once with no where clause at all.

Sitting at my desk at a previous company, there was a shout of "Oh SHIIIIIT!!!!!!" from the other side of the room. One of the DBAs had left his screen unlocked, and some of the others were larking about and had typed a drop tablespace command into his sqlplus session. He'd got back to his desk and had pressed return without thinking, rather than Ctrl-C.... I can't remember the details, but it didn't actually manage to trash the dat

Re:Dude

[RevMike]

DROP DATABASE;
UPDATE RESUME;

Re:Dude

[ceejayoz]

Ouch!

Ouch

[metlin]

Don't worry mate.

Look at it this way, when it's all done, it's a darned good excuse to get really, really drunk ;)

I'll have to start reading more carefully...

[TechnoLust]

I missed the JE where you started working for slashdot. ;-)

Sorry, didn't mean to kick you in the junk while you are down. :-)

Re:I'll have to start reading more carefully...

[FortKnox]

That's the difference, ya see...

I care about the users. That's why there is staging and testing platforms. Just one of those times a major bug slipped through all of them :-(

Don't worry...

[jawtheshark]

Happened to me this week too. I hosed the Permanent Orders in the WebBanking application in the last release. It was just a one-line fix, but hey... It was a coworker who found it, and it had been in production for the whole week already. No client complained ;-)

Anyways... you guys didn't have a holiday the 20th? (Ascention Day) I know that I was lying in the sun yesterday. Making the bridge today... looooong weekend. :-)))

not as bad

[subgeek]

it's not as bad, but i just inadvertently trashed a few hundred of our customer's special prices. i was tricky enough to do it in a way that i can't recover it. shit happens; it's not going to ruin this fine day for me, for it is friday.