> How does that sort of shit even happen? Who the hell promoted this person? Why?
A lot of the problem is the business side almost always knows nothing about the tech side or (bigger problem) who to trust and who is full of it. They're generally the sorts who can barely keep their own workstation under control. So you run into a few problems.
1) There's a shortage of qualified tech people who have been around long enough to avoid some of the common problems
2) The people ultimately responsible for hiring and business continuity don't know the difference between a good techie and someone who just fabricated most of their resume
3) Incompetent people tend to prefer to hire other incompetent people
4) Qualified people with experience can usually sniff out the hellholes during the interview process and GTFO
Specific to this company, they had extremely weak management. They had a profitable product that required very little actual engineering.. it was a mobile service but the app actually was self contained on the handset.. the only reason they hit our webservice was for billing and account related activities.
They hired the clown via contract from a consulting company to be their Windows/IT guy. Then their service was expanded to multiple carriers and they needed a real datacenter; because this guy was already there, they asked him if he had the ability to build out the DC. So suddenly this low tier windows guy has an opportunity to be an ops/infrastructure manager and add all sorts of fun stuff to his resume; he jumped on it. None of the people had any grownup level ops experience to evaluate his skillset or vet his plans/ideas.
After they had the DB meltdown I mentioned earlier they realized they needed a real ops person. The startup I was at was dying and a friend I worked with before (a business analyst SQL person) suggested me. They also wanted to put their product in teh clowdz; this was 2008-2009, the startup I was at was a pretty early AWS pioneer. (Pre-EBS, pre-ELB.. it was the wild west!).
Anyway I got in and it was a mess, I fought to help them clean stuff up and to avoid problems before they happened. The devs appreciated my efforts but due to the aforementioned weak management, the monkey dude was actually technically a manager? and I was just an engineer? Or something. So while technically I was "in charge" of production he still kept his claws in enough that he'd fight with me over every decision. I finally gave up and got a new gig when it became clear the situation wasn't going to change. Basically it came down to - I'm used to cleaning up peoples messes but I'm not going to fight for the right to clean up other peoples messes.
Ironically when I put in notice they actually offered to fire that dude on the spot if I'd stay. That made me doubly sure I made the right decision to leave; you should never put an employee in that situation to make that kinda choice. He was let go ~ 3 months later after they hired a replacement for me (they actually paid me contract to interview people for them). They're still around (and profitable!) despite themselves; most of the original people are gone due to a series of acquisitions.
Anyway I have a bunch of stories.. I've been doing internet startups for the last 18ish years. I like some aspects of them but sometimes I go park at a big company for a while to recharge. Maybe someday I'll write a book. I actually collect some of the funnier/more absurd stuff that goes on at the various companies. Mostly for my own personal amusement (and because other people wouldn't believe it otherwise). I anonymize them though.
A lot of the kids coming into syseng/devops/dev aren't necessarily bad, they just are inexperienced and have to learn the same lessons. That's the part that wears on me sometimes.. but if people are willing to take advice/mentoring from old timers(heh.. not even 40) then it's all good.
There has been a few times in my career I worked with lots of smart people (smarter than me) and it was a blast. Unfortunately that never seems to last all that long.