Comment My Biggest Booboo (Score 3, Funny) 898
It's the mid-90's, and I'm a developer with one of three vendors collaborating on providing a large US bank with a US-wide demographics database (a precursor to a modern CRM). The system was Unix-based, and pretty distributed for its time (8 front-ends, 8 back-ends, and 3 dedicated routing/replication servers in the middle).
It's the first morning of the pilot-to-production phase, and we're all sitting in the datacentre at our terminals, bringing the whole system online for the first time. I'm personally familiar with PC-based terminal emulators, not the fancy X-Windows stations that the client has on their premises. So, once we get everything finally up and running (and it's taken us about 2 1/2 years to get here from concept stage), I start exploring the settings on my X-Windows station. (Anybody remember CDE, and how... bizzare it is to configure, contrasted with KDE or Gnome?)
I'm fiddling around with settings, trying to create application shortcuts to fire up sessions with servers just the way I like, when at one point I get the message that a reboot is required for changes to take effect.
I issued the standard "sync; sync; shutdown -r now" command -- and just after I hit I realized that I had been typing into an xterm session ON ONE OF THE BACK-END SERVERS -- not the local X-station!
Well. The backend server goes down, and when the event-collector picks up the unavailability, it starts up alarms and red flashing lights (I kid you not), and also starts paging people (including myself, ironically).
I'm stunned, and terrified, for I've just brought down a system that had been operational for only 3 hours after being in development for 2 1/2 years.
We eventually get the server back up and running, and afterwards, the ProjMgr (from the prime vendor) drifts over to me and quietly mentions that I had a strange expression on my face earlier that day. We look at each other, and then he says it "must've been a s/w fault somewhere" before wandering off knowingly. (Whew!....)
Moral of the story #1: NEVER work in root/superuser accounts when you don't absolutely need to.
Moral #2: Use color-coded xterms to indicate which systems & what access-levels you are working with!
It's the first morning of the pilot-to-production phase, and we're all sitting in the datacentre at our terminals, bringing the whole system online for the first time. I'm personally familiar with PC-based terminal emulators, not the fancy X-Windows stations that the client has on their premises. So, once we get everything finally up and running (and it's taken us about 2 1/2 years to get here from concept stage), I start exploring the settings on my X-Windows station. (Anybody remember CDE, and how... bizzare it is to configure, contrasted with KDE or Gnome?)
I'm fiddling around with settings, trying to create application shortcuts to fire up sessions with servers just the way I like, when at one point I get the message that a reboot is required for changes to take effect.
I issued the standard "sync; sync; shutdown -r now" command -- and just after I hit I realized that I had been typing into an xterm session ON ONE OF THE BACK-END SERVERS -- not the local X-station!
Well. The backend server goes down, and when the event-collector picks up the unavailability, it starts up alarms and red flashing lights (I kid you not), and also starts paging people (including myself, ironically).
I'm stunned, and terrified, for I've just brought down a system that had been operational for only 3 hours after being in development for 2 1/2 years.
We eventually get the server back up and running, and afterwards, the ProjMgr (from the prime vendor) drifts over to me and quietly mentions that I had a strange expression on my face earlier that day. We look at each other, and then he says it "must've been a s/w fault somewhere" before wandering off knowingly. (Whew!....)
Moral of the story #1: NEVER work in root/superuser accounts when you don't absolutely need to.
Moral #2: Use color-coded xterms to indicate which systems & what access-levels you are working with!
