IT manager meddles in backups; poo storm ensues (Score 1)

We had an "IT executive" panic one weekend ...

"Storage team" was moving from Vendor N to Vendor E. Lots of files on Vendor N arrays. Users made heavy use of WIndows/Unix cross links. Storage "wizards" were completely unaware; did almost no due diligence.

Move starts on a holiday weekend before major customer release was due. Wizards unaware that users were changing files. Wizards do their backup from N and restore to E, place E in service. Engineering goes nuts, nothing will build, can't find their files (links broken). Wizards throw up their hands, saying "we didn't break anything".

Of course, what was backed up was old by now. "IT executive" (BTW, didn't even finish his business degree, much less any technical degree) decrees: "PUT IT BACK THE WAY IT WAS". OK, great, now the builds (sort of) work but changes made on Saturday and Sunday were lost, so what is built is not up to date. "Executive" and "wizards" throw up their hands again, saying "this is application specific data, we don't know how to fix it".

Actually, it's just Unix files, so if they had been inclined to help, they could have. However, they were not. Phone call to my boss on Sunday night, explaining only as much as needed to indicate the "wizards" were basically throwing this in the user's lap.

Cue me and two other guys picking through the changes and restoring consistency to the files, based on priority needed to make customer delivery. We wrote a bunch of scripts to figure out the mess laid on us by "storage wizards" (who kept saying "we don't write scripts, we maintain your storage"). These guys were completely incapable of sorting through their own mess.

The mess was a mass of directories and two partial "backups" (each directory) which were inconsistent from what the "wizards" said they did. They couldn't tell the same story twice in a row, and had no timeline of what happened when.

They screwed up routine backups for at least 3 years until someone challenged them to test them, then they stalled for 2 more years fixing the problem. The switch from N to E was supposed to be completed in one year; it took two because E didn't have the link feature that users had exploited in the N product. This was supposed to save the company tens of thousands of dollars. Rather, they ran BOTH the N and E equipment for a whole year while "wizards" had to learn how to make E equipment do what N was doing before.

Executive and "storage wizards" keep their jobs for years following.