Noryungi's Journal: How to make HP Tech Support laugh. Nervously.

Since I am a godless atheist, I'll swear on the holy names of both J.R. "Bob" Dobbs and Eris that the following is absolutely, 100% true.

Only the names have been changed to protect the guilty. And my employment.

From: DumbIdiot <[e-mail]>
To: [e-mail], [e-mail], [e-mail]
Cc: [e-mail]
Subject: I think I made a mistake on galactus.
 
Hi,
 
I was connected as root on galactus, and I entered the command rm * in /
 
I though that I was in /home/programmers/dev/3.14/backup instead.
 
Is this bad?
 
--
Clueless DumbIdiot.

Yes, gentle reader, the above email is (almost) 100% true.

If you understand (heck, this is Slashdot, you should understand it) what "rm *" does, as root, in the / directory of a UNIX machine, you are probably already screaming: "Oh, the humanity!".

But WAIT! There is more to come. The horror has just started.

From: [e-mail]
To: [e-mail], [e-mail], [e-mail]
Cc: [e-mail]
Subject: Re: I think I made a mistake on galactus.
 
Nah, it's OK.
 
Galactus is a Tru64 machine, and these machine have nothing important under /
only a few login shells.
 
UNIX admins: could you restore the shells? Or just copy them from Kraken,
it's another Tru64 machine.
 
--
Manager1

Yes, I told you so, gentle reader. The horror has just started.

Please note that the anonymous "Manager1" above is not just any manager.

No, he is the head of the programming team at "ch.megacorp.com". Most of his work day is spent on UNIX machines. He is the kind of guy who is supposed to have a clue , for Pete's sake!

On the other hand, he is the one who insists on giving all his programmers the root password on all the machines, because, well, because "they need it to update the software".

Anyway, I tried to compose as polite a reply as I could. Here is what I came up with:

From: [e-mail]
To: [e-mail], [e-mail], [e-mail], [e-mail]
Cc: [e-mail]
Subject: Re: Re: I think I made a mistake on galactus.
 
Hi,
 
Actually, Tru64 stores the kernel and a lot of system information in /
 
For example, on a Tru64 machine we have here, I have this:
 
me@sauron$ ls -alF /*unix*
-rwxr-xr-x 1 root system 12887904 Sep 29 2003 /genvmunix*
-rwxr-xr-x 1 root system 18784112 Mar 29 2004 /vmunix*
-rwxr-xr-x 1 root system 18357024 Mar 29 2004 /vmunix.PrePatch*
 
So, I think your machine may well crash pretty soon.
 
--
me (just a UNIX admin doing his job, ma'am).

Yes, my friends, my situation was already dire.

But it goes without saying that a good deed cannot go unpunished:

From: [e-mail]
To: [e-mail], [e-mail], [e-mail], [e-mail], (longest list of managers you have ever seen)
Cc: [e-mail], [e-mail]
Subject: THIS IS CRITICAL!!! Re: Re: Re: I think I made a mistake on galactus.
 
People,
 
Galactus is a CRITICAL MACHINE. IT CANNOT BE ALLOWED TO FAIL.
 
I don't think [e-mail] understands the seriousness of the situation:
galactus could EXPLODE at ANY MOMENT and we HAVE to deliver a new version of the
software to our clients TOMORROW!!!
 
THIS HAS TO BE FIXED NOW!!!
 
--
Manager1

OK, it can really get any worse than that... Right?

Wrong.

From: [e-mail]
To: [e-mail], [e-mail], [e-mail], [e-mail], (longest list of managers you have ever seen)
Cc: [e-mail], [e-mail]
Subject: Re: THIS IS CRITICAL!!! Re: Re: Re: I think I made a mistake on galactus.
 
Sorry,
 
I just tried to connect to galactus and here is the reply from the machine:
 
$ ssh dumbidiot@galactus
SSH: Connection timed out.
 
Is this bad?
 
--
Clueless DumbIdiot

Oooops. The "critical" machine, that "must not fail"... Just did.

Yes, sirree, it crashed.

This server is no more! It has ceased to be! It has expired and gone to meet 'is maker! It is a stiff! Bereft of life, it rests in peace! (Etc... etc...)

And here is why I am still at work right now (8:30pm), trying very hard to restore this machine instead of being home to kiss my wife and kids.

And here is also why, gentle reader, I can make every single person at HP European Tech Support laugh. Nervously. Some of these techies could not believe it. Some believed it. But they all laughed.

Now, if something like this happens to you, here are two solutions:

If your machine is still on (and the machine renamed as "galactus" in the emails above stayed on for a couple of hours before crashing):

There is a copy of the Tru64 kernel in /usr/sys/MACHINE_NAME/vmunix.sys: copy this in the / of your main disk, and you should be OK. Theoretically.

If your machine crashed... Remember that /usr/sbin/advscan is your friend. And not just your friend, but your best friend. Your sugar daddy. And not just your sugar daddy, but your saviour. Trust me: it truly is THAT great.

Now, I hope my tape restore has fnished while I was typing this... Then again, probably not.

Nope, tar is still sitting there...