Tet's Journal: Learn to do your job properly

So we've bought this credit card company. They've got over a million cardholders, and around a billion pounds a year turnover. We're in the process of migrating their IT systems from their previous owner to where we're managing and running them. Not an easy task, but we've got a deadline to hit (if we don't get them migrated by the end of this month, we've got to pay half a million per month for their previous owners to continue providing IT services).

So last weekend was email. We'd come up with a bulletproof plan, with 7 steps, each of which needed to be followed in order to avoid breaking everything. Step 1 -- change MX to our mail server, which to start with, is configured to just forward mail for that domain onto the previous MX. That lets the MX propagate through DNS without affecting service. The message still ends up at the same server, whether it uses the old or new MX value. Simple. So, I changed the MX on Friday evening, and lo and behold, the fuckwit email admin at $PREVIOUS_OWNER had configured their Exchange server to stop accepting mail for that domain. Hello? Reality calling... you need to keep accepting email for that domain. Like it says on the plan. Like we discussed in the conference call we had on Friday morning, so everyone was crystal clear about what they had to do and when. Like not until you get to step 7. So you're doing it at step 2 why? Just to piss me off and waste 9 hours of my time on a Saturday?

It turns out that "well, we're not the MX for that domain any more -- we can't accept mail for it, otherwise we'll have an email loop". Moron. I mean OK, so he's using Exchange. But you'd think that being an email admin he'd bother to learn how email works. Like the difference between local delivery and relaying. Like how and when an MX record is consulted. Like the difference between an envelope and the message headers in the body. So as it turned out, they refused to backout the change, and a billion pound credit card business was without email for 22 hours over their busiest weekend of the year. They still haven't backed out the change. But we managed to work around it by finding another one of their mail servers that was still accepting mail for that domain, and forwarding to there. Sure, I get to charge a fortune in callout overtime. But I'd rather have been spending my time relaxing at home...

I've found that

[ces]

what the average Exchange "administrator" doesn't know about what they are actually doing could just about fill the Grand Canyon.

I think the bad reputation Exchange gets is mostly due to 2 things:
1) most exchange admins don't know what the fuck they are doing.
2) most exchange admins can't resist "tweaking" things like obscure dialog box settings, registry entries, and raw active directory entries especially ones that can't be backed out easily dispite not having any damn good reason for doing so nor unders