Martin Blank's Journal: Some nights just suck

I think I mentioned before that I work nights. This is usually not a bad situation, as it's quiet, I get to explore a little, and I've been mildly anti-social (some would say psychotic) the last few weeks anyway. However, I'm seriously on edge now. The NOC pager just went off on one of its daily notifications that the news is in, and it took some willpower to remain seated and not throw the pager across the room.

Here's the gist of it. I work for a company that hosts servers. Sometimes we manage the boxes, and sometimes we just provide power. One of our customers in the former set maintains about two dozen boxes here. Over the course of a few hours, they ALL went down. Argh. Ack. Fook.

I suggested the problem was related to routing changes made by another company with whom we work to some small degree. Our router specialist, freshly roused from sleep, swore that the two problems had nothing to do with each other. I struggled for the next five hours to determine the problem before finally bringing my boss into things.

Guess what? It had to do with the routing changes.

Specifically, we lost access to a domain controller, which authenticates the boxes to each other. Sometimes I like it when I'm right, and sometimes I want to knock my head into a wall. If I had just CHECKED THE DAMNED LOGS when the whole thing started (like I've been doing on every other problem so far), we could have seen the authentication errors and looked at this much sooner.

Because of my delayed response, and because of a few errors in how I've handled calls, I've been getting some advice on how to do some of these things (properly handling these issues from a customer standpoint). I really do deserve it, even if it is a little patronizing. It was lame to wait so long, and still more lame to not check the logs.

Live and learn, I guess. And sleep, I hope.