The East African SEACOM cable has been having outages lately; they posted an outage notification due to a cable break off the Egyptian coast at 08:40 UTC yesterday (March 27th, 2013). Of course, this has been having knock-on effects: for instance, many South African ISPs use this cable as their primary international link, and have had to fall over to secondary links resulting in significant service degradation.

Co-incidence? Perhaps, perhaps not...

...Slashdot reported on a 25 year old BSD bug being resolved back in May 2008.

And these are just the ones we know about -- there may be yet older bugs (particularly in proprietary, closed-source systems, where the source cannot be reviewed by the general community).

Thing is, it was a small, young company still figuring out what the best practices were to follow. When I had started there, things were extremely ad-hoc: there was pretty much no process at all (not too dissimilar from this, actually!), and I knew pretty much nothing when I started. Towards the end of my stay, I had pretty much taught myself concepts such as proper source control, process models and that kind of thing, and was trying to get things implemented (despite resistance from the "greybeards"). At the time when all this was going on, things were moving in the right direction, but the whole system was still far from perfect. It's likely been sorted out since.

Like I said, sometimes it's necessary to learn the hard way.

One of my previous employers, a while back, employed an individual who I will henceforth refer to as the Office Freak From Hell (it had various freaky habits: no personal hygiene, odd behavioural patterns, that kind of thing). I kind of ignored it at first (except to avoid it as much as possible), until it was moved over to my team. It didn't take me long to realise how useless it was -- his code was often delivered late, and was always of a poor quality (example: using strings as every variable type -- really, what the FUCK?). Between my manager and myself, we tried to mentor him, correct him and all of that -- we couldn't fire him straight away as South Africa has really fucking stupid labour laws which makes firing a tedious and difficult process at best (and you'd better not slip up, otherwise the fucktard can successfully sue for damages and the old position back). Meanwhile, I was searching for alternative employment (although mainly because software development in Durban is a dead-end industry, the OFFH was a major contributing factor), received an offer that I couldn't refuse from a company in Cape Town, and put in my resignation. I still had to work a calendar month's notice period though (Americans, things work differently over here!).

That's when things got interesting.

My manager and I started the process of handing over all my projects -- most to the rest of my team, but a few went to the OFFH. It didn't take long for the OFFH to piss off one of my soon to be ex-clients to the extent where top level management got involved, the OFFH was finally pulled into a disciplinary hearing (wasn't fired, but received a final written warning), and I had to step back in and clean out the mess. The next day, the OFFH put in for leave on the Friday coming up, went away... and never came back. It was formally dismissed for absconding shortly afterwards.

That's when we found what was really going on. To summarise:

• - The code that would be pushed through to production was often not the same code checked into the source code repository, and the production code was riddled with security holes, backdoors, and that kind of thing. (Since I used the code in the repos for code review purposes, I never picked this up.) A few months after I'd worked my notice period and left, I heard that they ended up writing new, parallel systems and chucking everything he'd worked on, while doing their best to maintain it until the parallel system was complete. (Side note: I left on friendly terms, and I still keep in contact with those guys.)
• - When we went to try to get source code from his machine (see point above regarding the source repos), we discovered a whole lot of background services constantly maxxing out the CPU. We never found out exactly what they did, but given other discoveries, this pretty much resulted in the network team dropping everything and performing a full security audit of absolutely everything.
• - He would often tag in after hours and during weekends. I remain convinced that he was up to absolutely no good during this time, particularly as I am in possession of an IRC log detailing an intrusion he was involved with on the South African XBox 360 fansite around mid-2009.

So, while we thought we were dealing with mere incompetence, in truth, the OFFH was a malevolent fucktard.

All of us involved has learned our lessons -- personally, I'm far more security conscious, and the folks I worked with are far stricter regarding who they hire, development practices and policies, and that kind of thing. As for the OFFH, it seems to have vanished into thin air...