Iamthefallen's Journal: Sysadmin Excitement!

Came in this morning and went ahead with regular duties. Nothing spectacular going on so I went into the bosses office to talk about some reports I'll be developing in the near future. Our full time sysadmin comes in and tells us we have a problem. Two drives failed on the production database server, 3 minutes apart, on separate buses. One on the data partition, one on the system partition. The HP management tools never picked up on it, and in fact still report that everything is just peachy.

We go back to the server room to swap drives out when we hear an odd noise, look back and another drive has failed on the application development test server.

Ruh-roh.

I know what your problem is

[IamBMETammy]

Goblins. Small vicious little buggers probably spent all weekend sabotaging all of your drives. I recommend some brownies, they attack your goblin nests, destroy the eggs, and drive off the adults. Now brownies can cause problems as well, but that's where the sprites come in. Just release a few and they'll knock out your brownies. Be sure to use European sprites, none of those pesky Japanese yokai. Now the sprites should die out on their own since technology and sprites don't really mix, but if they star

So...

[ellem]

When do the new Boxes show up?

Your backup vendor called

[Degrees]

They want you to know they are looking forward to the midnight meeting tonight. And yes, in the fine print, it does say that you were supposed to be running cleaning tapes all this time....

Re:

[Iamthefallen]

Actually, I think we're good now. We moved some drives around, rebuilt them, and kicked it a few times, and it looks good so far.
Now we just have to figure out A, WTF happened. B, Why did the management software never pick up on it. C, Why didn't the tool that did pick up on it send out the DANGER DANGER WILL ROBINSON! email.

We think it was just a glitch in the controllers, and luckily we had just one disk in each set fail rather than two on the same.

Mai Too Sense

[BandwidthHog]

While this isn’t exactly my area of expertise, I don’t believe in coincidence when troubleshooting. Unless you’ve found something that ties all three(!) failures together, then you’re merely treating the symptoms and should expect the new drives to fail at any moment.