When I was a junior programmer working on a mainframe, I was given a problem ticket for an intermittent issue. I stuck diagnostics into the code, but because my disk quota was far to small, I sent the output to a virtual printer that I looped back to my account. Unfortunately, after I got the whole testcase set up (couple hours) the mainframe crashed and I went for coffee along with the rest of the 300 users on the system, for the 10 mins it took to restart. After several days where I hadn't been able to make progress because of the suddenly frequent mainframe crashes, I got a message from the operator asking me to delete my large spool files, since the mainframe was crashing due to a lack of spool space. That's when the penny dropped that my testcase had been exhausting the system spool space, crashing the mainframe about 8 times. Probably $100,000 in lost labour.


Years later, working on extending some high reliability software, I found some bugs in pre-existing code. The system had some internal checks and watchdog timers that would force a restart if it thought some code was taking too long. Both bugs would trigger the restart system by making something take too long and triggering the watchdog timer. One was in very complicated code, but explained some intermittent issues we'd seen over the years. The other was in a newly released, still unused utility, that didn't work properly on old HW, but would need to be re-written to fix. I only had time to fix and test one bug before going on a month long vacation, so I fixed the complicated one. While I was on vacation, an alpha release of the product went out, and promptly started crashing intermittently with stack corruption issues. I got back, to find six such tickets on my desk. In the meantime, the broken utility had acquired some users, so I decided to spend a couple of days fixing the utility.

It turned out that the stack corruption issue was holding up the production release, worth many millions of dollars.

Of course, I wasn't able to reproduce the intermittent stack corruption.

I spent 3 weeks looking everywhere, trying anything to reproduce it, resorting to rebuilding the alpha load where I could sometimes reproduce it, but not if I loaded my diagnostics.

Meanwhile, management was getting very antsy about the revenue implications.

My boss was very good, and sheilded me from the flames, but I didn't like seeing him getting fried, as the release date kept getting pushed.

I tried hunting around to see if anyone had been changing code in that area of the system, but of course, there were only my updates. I asked anyone I could find for suggestions, and nobody had any ideas until one person said it reminded them of one very old issue they'd worked on, and described the problem they'd had.

I went back and checked my archived output. Sure enough, I'd been a bit careless testing the broken utility before fixing it. I only checked that my testcase triggered a restart, not why. It turned out that long before it could trigger the watchdog timer, the utility corrupted the stacks of other processes.

I'd just spent 3 weeks holding up an important release, because I didn't realize I'd already fixed the bug.

Best hardware hack was one a former boss did, in the late seventies. The building housing the computer center had caught fire, and everyone needed to evacuate the building he had dozens and dozens of backup tapes, but knew he couldn't cary that many out in his arms. The he realized he could string most of the 9-tracks onto a stand-alone coat-rack, and so he used that to carry them out.

A friend of mine in Canada started a website I'll call xyz for his own use with some useful info published daily in a graphic I'll call pqr This was around 2000 At that time he searched thoroughly, and found nothing called xyz or pqr.. Pretty soon other people found the site, and started asking to donate, because they liked it so much. He managed to get enough donors to cover registration with enough left over for some nice gear every few years. Unbeknownst to him, around 2003, someone else created a startup I'll cal pqs. They registered pqs as a trademark quietly in the state where they were located. After the 5 years wait time was up, they were granted the trademark. At this point, the startup was still vapourware, but they were able to pitch to a VC. The VC demanded that the trademark be defended, so they sent cease-and-desist letters to everyone with an even remotely similar name.Pqr was "too similar" to pqs, so my friend got a letter. One of his many donors was a lawyer, who took his case probono. After much negotiation - including that my friend's use preceeded the trademark, the lawyer eventually told my friend that it was somewhat likely he'd win in court, but he'd have to travel to that state basically at the whim of the VC who'd be sure to arrange adjornments until it was inconvenient for my friend, that it would probably take years and $300,000 to actually win, or he could get a licence for $5,000/year, or change the name. My friend changed the name. But still quietly snickers when people still call it pqr.

That might be conventional wisdom, but it turns out to be wrong. Not only will the employee feel annoyed, but everyone of his co-workers will see that management secretly sees them as garbage the instant something changes. I was a low level manager at a large company that did layoffs. At our How-to-lay-people-off training session, the HR person told us that the people being laid off would continue to have full access unless we had good reason to suspect an individual might try sabotage. We were surprised, and asked about this. The HR rep said that they used to boot people right out the door, with security packing their belongings in a box. And they got a lot of lawsuits. Someone did the calculations, and figured out that even if departing employees totally cleaned out all the office supplies, it would still be much cheaper than the lawsuits. Important information was all backed up anyways. The extra bonus, was that treating departing people humanely not only meant they were willing to come back after things picked up again, but everyone else in the office felt much better about the bad situation.

It is too easy to dismiss critcism as whining, just because the critic does not have the solution for you, or does not express themselves politely. Consider this piece of experience I hope none of you ever have to live through: I worked for a company with a large SW system that had a major architectural wart, that was required to meet real-time. performance requirements. For various reasons, the company decided to re-write the entire system in an OO language. I went to one of the early presentations on the new system, and observed that the new architecture had done away with the wart because it caused too many maintainability issues. I stuck my hand up to ask just how the system performance could be maintained without the wart. The answers ranged from the need to get it working before tuning it for performance, to the idea that Moores law on the HW side would solve the problem. After the presentation, the presenter/lead architect suggested that I join the team to "help them improve". However, it was clear that real-time performance was going to be totally ignored until later, if ever. Since this was a major criteria for the product, I declined, and suggested they research in more detail why the wart had been created in the first place. Because I declined to join the project, and waved the red flag about performance in public, the very influential lead architect labeled me an unconstructive critic and whiner. I changed jobs to an area outside the architect's influence. Several years later, they were finally able to deliver a system, whose first inital transactions were 5000x slower than the original system. When it became clear to the senior mgmt that this unacceptable degradation couldn't be fixed through optimization, the project was cancelled with a $1 Billion write down. And not long after, the company went broke. I later learned that I hadn't been the only one to raise concerns about performance. And every single person who had spoken up had been dismissed as an unconstructive whiner, because they didn't have the solution, or "were too stupid to understand the architectural vision". It can be hard -- very hard -- to hear criticism about your dream/idea. It is very easy to label the critic as a whiner. Not having a solution to an issue does't mean it isn't an issue. The measure of any organization, is how it deals with dissent. If someone is annoying you, you are too close to be objective as to whether they should be deleted. Maybe we need new roles, for non-invested people who listen carefully to "whiners", and translate whines into de-personalized issues. Even sacred cows should periodically be taken out and inspected.

I have friend who's a slightly computerhobic musician. They were so proud about learning how to synthesize on their computer, that they decided to try making a YouTube video. They spent hours peerfecting some classical piece (Handel, I think). They created a YouTube account, figured out how to put the mp3 to a static picture, and posted it privately, intending to figure out how to animate the music score. Before they had the chance, (and while the video was still private), they got a takedown notice. They were totally in a panic that this could impact their day job. I helped them put together a counter notice. When they got the demand to "prove" they owned the content, there was much more panic. Even though the-powers-that-be took the notice away, there was nobody they couldd call about the notice, nobody but me for them to yell at about how unfair it was.

Upshot: to this day, they've never gone back to finish that video, and publish it. And if you talk to them about synthesizing music, instead of happy pride, you hear panicy shuddering and unhappiness.

I think there's a serious imbalance of power when legitimate owners have to prove their innocence, and the spawners of that notice get off with no consequences. How do we fix this?

I look for an app to do something I want. I find umpteen poorly described ones that seem like they might fit the bill. I try three or four at a time. Despite a promising description, they don't really do what I want/crash/require too much set up/they forget all the options I laboriously set up last time/they demand inexplicable permissions. I delete them, and try another three or four or five. Eventually, by luck I find an app intended for a different purpose, in a different category that does more or less what I wanted. If I'm lucky, it doesn't nag me to give good feedback until I can't stand it, stop using it, and look for a better alternative.