How To Make Software Projects Fail

Bob Abooey writes: "SoftwareMarketSolution has an interesting interview of Joel Spolsky, of Joel on Software fame. Joel, a former programmer at Microsoft, discusses some of the reasons he thinks some very popular software companies or projects fail, including Netscape, Lotus 123, Borland, etc." This interview brings out some mild boiler-room stories which sound like they could be the basis of a good book, along the lines of Soul of a New Machine .

Re:Isn't it obvious?

Actually, reading this interview shows how there where serious blunders performed by NS, Borland, etc. In each case, while MS improved their software, the other companies rewrote their software.

Re:Isn't it obvious?

It is funny how every company he talks about lost to MS. Seriously though, one of the things he does say is:

Fortunately for Microsoft, they did this with parallel teams, and had never stopped working on the old code base, so they had something to ship, making it merely a financial disaster, not a strategic one.

IOW, have more money than God and throw it at any problem you're having trouble with. The minnows in the pond get beaten up by the 800lb gorilla (or something).

Re:Isn't it obvious?

> IOW, have more money than God and throw it at any problem you're having trouble with

Didn't work for IBM in the early 90's, didn't work for Detroit in the late 70's and early 80's, still doesn't work for the government.

Re:Isn't it obvious?

Show me a government owned power company in the USA.

OK, I'll mention a few: TVA (i.e., Tennesse Valley Authority, which lifted an entire region out of utter abject poverty during the Depression, SoCal's DWP (which not only distributes water and generates power, but also manages to generate power while distributing water), Sacramento's Municipal Utility District (MUD) which generates and distributes most of the power in north-central CA, and finally the BPA (i.e., Bonneville Power Administration) which built and still operates most of the hydroelectric power generation and transmission in the Columbia watershed. The Northwest has a lower cost of living partly due to low power costs (though it isn't guaranteed and it has been rising) and low water costs (likely to continue given near term global warming effects). Water, Power (and soon, Broadband) are _exactly_ the infrastructure investments that our government does well and should control. Private utilities are very vulnerable to economic fluctuations where their executives' self-interest leads them to try foolish deals and daft accounting tricks in search of short-term performance, while government can weather tough quarters (and years) without worrying about the stock analysts.

In case you hadn't noticed, the major private power utilities in California are in bankruptcy and are desperately beseeching the State to bail them out (and might yet stick the tax and rate paying citizens after all, given how cozy their lobbyists have been with the CA PUC, Legislature, and Executive branch fixers, just about forever). One can only hope that the CA government and regulators now realize that the public is watching with interest and will nail them if they screw it up further, so they might fix it properly.

Of course, private utility executives and board members never do get held accountable, nor do their government co-conspirators, but if things were to be really just, there'd be a few of them hung from lamp-posts in San Francisco before this is over. Screwing the public for private gain is just the sort of thing that deserves "extreme prejudice."

Government utilities are a good thing, mostly (WPPS notwithstanding, but that was a _private_ boondoggle admittedly triggered by a BPA error). Private utilities are simply disasters waiting to ripen, explode, and be discovered, unless they are regulated into castrated quasi-governmental entities. The term "private utility" really is an oxymoron.

Perhaps you should read the article

Before you jump to false conclusions. A lot of companies that Microsoft "drove out of business" were driven out of business because they made stupid mistakes. Yes, MS's money and marketing helped, but some of the stupid things these companies do is their own damned fault.

You may want to check out this article by Robert Cringely: Microsoft's C# Language Might Be the Death of Java, but Sun's the One to Blame [pbs.org].

A lot of truth in that...

Re:Perhaps you should read the article

The interesting thing is that Microsoft made plenty of stupid mistakes too, but since they were powered by monopoly profits in OSes (and earlier on by licences for BASIC in ROMs), they could afford to wait out their mistakes and just keep throwing money at the problems until they straightened them out. As long as a company is successful in the long run, people forget most if not all of their stupid mistakes, but if the stupid mistakes take down the company then people remember it. The history books are written by the victorious (in the short term at least), so I would take a lot of this guy's story with a grain of salt, as it is certainly far from unbiased.

It is a pretty bad situation for a market to be in if any one company is so big that all they have to do is wait for their competitors to make a mistake in order to be able to crush them. When any one company wields so much power, it makes it nearly impossible to sustain any sort of competition. Not to mention that when a market is ruled by an 800lb gorilla, all of the smaller players are pretty much forced to take more risks and make other decisions differently than companies do in a market where there are at least two or three players splitting up significant chunks of market share. Sometimes those risks pay off brilliantly, sometimes they are stupid mistakes.

Re:Perhaps you should read the article

Microsoft has had a monopoly position for a long time. Most people don't remember when there wasn't a such thing as an IBM PC or MS-DOS. Microsoft got into a monopoly position with MS-DOS pretty quickly after its release in 1981. Within a couple of years it had killed off CP/M and the 8 bit 65xx OSes. So certainly since at least 1984 or so Microsoft has basically had a monopoly position in OSes.

But wait... that wasn't Microsoft's first monopoly position. Even before MS-DOS they basically had a lock-hold on BASIC interpreters, which were one of the most critical parts of the pre-IBM PC desktop. Apple's BASIC, Commodore's, and Radio Shack's were all licensed from Microsoft. Most CP/M machines also bundled Microsoft BASIC. In fact a strong case could be made that the MS-DOS monopoly which grew into the Windows monopoly was itself leveraged from the BASIC monopoly.

There is a difference between pointing to monopoly power as the primary reason for Microsoft's success than the only reason. Saying that Microsoft's monopoly power had nothing to do with the failures of other companies is as wrong as saying that those other companies made no mistakes.

As for Microsoft's marketing, I am not so sure I would call it 'shrewd' as pervasive and persistant. They've outspent just about everyone else for years, with the possible exception of IBM, but that is easy when you have monopoly profits to fall back on. It would be hard for a startup to outspend Microsoft on advertising, even in a niche.

Good point

He says:

"My theory is that this happens because it's harder to read code than to write it."

He couldn't be more right. I've recently been asked to port some code from another group in the company. Upon first reading it, I found global variables being referenced from everywhere, and it looked terrible.

The more I looked at it though, the easier it got to read, and having an existing code base to work from made things much easier.

Plus, when I have problems with it, I can blame it on a "design error" by the previous programmers!

Re:Good point

This is why it's important to force your developers to (gasp) comment their code. Of course, 99 times out of 100, this won't happen because either (1) the boss thinks that'll slow you down and you'll miss your release date or (2) your boss has never written a line of code in his life and doesn't even know you can comment on that computer codes thing.

Re:Good point

Good code speaks for itself about what it does, but not about WHY it does something and that's were comments come in handy.

Re:Good point

I couldn't agree more. In a similar vein, I removed the turn signals from my car. I get .0000047% improved performance and, after all, what good are signals? I know where I'm going.

Re:Good point

If you think no comment is ever useful on well written code, you shouldn't be a developer. If someone new has to come into your project (especially a jr. engineer), well placed comments will greatly decrease the time required to understand the overall function of your code. By assuming your code speaks for itself, you're assuming everyone is exactly as familiar with the code as you. This basically forces a newcomer to study your code down to every nook and cranny before it even makes sense. Not commenting your code is nothing other than lack of discipline and a sign of indifference towards the overall success of the project.

or (3)..

or (3), incessantly repeated nerdisms such as "if it was hard to write it should be hard to read" instill an improper sense into young, impressionable programmers.

Re:Good point

Good code is not just code that compiles and runs efficiently. Good code also has the following properties:

• Clear, Consistant Formatting - This code complies with the company standard for writing code. Indents are properly nested, Functions are named consistantly, variables use Hungarian Notation or some other standard. Any programmer should be able to look at code by another programmer and pick up on it very quickly, without shaking their head and saying "What the hell were they thinking?"
• Copious Comments - Lots of comments, clearly written and explanatory. What does this function do? Put a block at the beginning explaining it. How does this algorithm work briefly? Write a paragraph if you have to. The best comment I heard was from a friend about a former coworkers code: "It's English with some C++ thrown inbetween the comments."
• Documentation - Anyone who shrugs this off is an idiot. You always have time for documentation. And it's not just for the instance where a programmer gets "hit by a bus." It's for people who leave behind code when they quit, or go to a new project. It's for the new hires, so they can understand and study and learn good design, good techniques, and developer rationale. It forces developers to explain themselves. And it allows non-techies to understand what they're doing. Imagine you had to get through 12 years of grade school with no books. Pretty frightening, eh? Documentation is good. Write it.

Coders who follow these rules truly are an asset to their company. Geeks who hack, write unreadable code, and utter geek credos about enforcing obfuscation and being purposefully vague have no place in a business environment.

Hubris, laziness, and impatience

Let's compare:

for (i = 0; i array_size; i++)
free(array[i]);
free(array);

and now let's look at:

// get rid of the array
for (i = 0; i array_size; i++)
free(array[i]);
free(array);

Has your life *really* been so harmed? Is this *really* so terrible? Comments should not be written with the thought that your university professor would know what everything else means. Comments should be written so that all of those folks without a PhD in CompSci. know what it means.

What if the next joe to hit your code doesn't have a degree? What if the recently-hired intern was just handed a "C in 21 days" book and told by the manager to "fix it" because the programming team is snowed in (or similarly unavailable) and the customer is screaming? (Yeah, try and tell me that's never happened...)

A fine use of comments is (for example) every ten lines to say, in general, what is going on. One thing I used to do is write a comment at least every 10-15 lines. Why? When the next joe who comes along has to read/edit my code, scanning through some periodically placed comments will *always* be quicker and easier than reading the code. ...assuming they are English speakers of course. Proof? I have programmed (I don't count the BASIC years) for ~10 years. I have been writing/speaking English for over twice that long. Which do you think I'm better at?

The code effectively shows my implementation, but may not show my intention. I have coded for years. I started dreaming in code several years ago. Shortly thereafter, the code actually worked when I typed it in the next morning. That isn't the point. How good a coder you are isn't the point.

When you have a hundred thousand lines of code to go through, comments become like "Cliff's Notes." For the quick patch (probably the majority of code being written by most people), comments are invaluable. Who cares if I didn't read Moby Dick if I can still pass the pop quiz? If I need to make an indepth study, I can still do this, but thank god for the "Cliff's Notes."

Now then, on to the "proper" use of comments.

1. Write out what you are planning to do in English. (or whatever else may be the dominant language in your development group) Fill in every step in the problem. This is NOT psuedo code. This is akin to: Find out who www.yahoo.com is, open a connection, ask for the main page, and check to see if our cache is still valid. If the cache is stale (the yahoo page has been updated), get a new copy of the main page. If the cache is still valid, pull the page from cache instead. Drop the page into the "ready" bin and send a message to the user that the page is here.

2. Make a copy and label it "documentation."

3. Go back to the original, fill in all of the logic in whatever programming language at the appropriate points in your "documentation," and label it "source file."

This means that your documentation is done, your code is adequately commented, and your algorithm and intent(!) are clearly defined for both your co-workers (and yourself when you have to fix something ten months from now). If you can't spell out the problem and the solution in your primary native language, you sure as hell better not be trying to spell it out in a programming language that members of your team have only been using for two years.

The only excuse not to do the above is laziness. For some people, laziness is not considered a bad thing. It was noted as being one of the main virtues of a hacker -- hubris, laziness, and impatience. Hell, according to this measure, I myself am lazy from time to time. But cut the bravado, the beating of the chest, the battle cries of "I'm smart enough to figure this out, so should you be," and call a spade a spade. Avoiding comments means that you are being lazy.

Re:Good point

From a purely intrinsic point of view, I agree with you StaticEngine. From a purely practical point of view, I couldn't disagree more.

Let me explain myself. I have been the type of programmer you speak of. I have written copiously commented code. I have properly formatted my code and used standardized function names and such. After all, I was taught in college to write and comment my code so that any programmer could walk in off the street and understand it easily; that made it easy to replace me and I was.

It seems that when you follow good programming practice, you end up destroying your job security; and as silly as it sounds... it appears to be sooth.
Jaded in a realistic world.

Job security?

Dude, think about what you are saying. Do you want to keep maintaining your old crappy code or pass that job onto someone else? Or do you want to go write some new code?

Your perspective assumes your company requires a fixed amount of software. Think more imaginatively.

Better documentation means you can shove maintenance to a more junior programmer with less pushback.

Also, without good documentation, its a b*tch to try to outsource/handoff pieces of the code you don't want to bother writing.

Besides, I don't care how well documented your code is, you should always be able to convince a boss that its more efficient for you to make changes to it (even at higher salary) than some cheaper guy who has never seen the code before.

--LP

Re:Good point NOT

Coders who follow these rules produce what? 3000-6000 lines of code a year? Ain't gonna get product out the door that way. What you get is code with lots of comments about what it is supposed to do, but doesn't because of all the time spent on documentation instead of design and debugging.

Excellent point. My philosophy about commercial software is this:

Never forget why you're writing the code to begin with. The point is to get working, stable code out the door as fast as possible. Anything that does not accomplish this is a waste of time.

Doing your architecture work is fine, but do it on a whiteboard in your cube with your co-workers. Don't waste time holding formal design meetings and drafting useless documentation and diagrams because, honestly, nobody will ever read them.

Modularize/componentize your code as much as possible. Document what the module as a whole does and what data it requires and what data it returns . You shouldn't have to waste time commenting every single peice of code. If you''ve modularized and documented what the module does, any decent programmer can figure out the rest.

In addition to not hiring idiots, don't hire people who love to wallow in bureaucracy and process. Besides not getting jack shit done, they impede everybody else.

If you want to comment and spend hours drawing Visio diagrams, fine. Wait until after the product is released to do this. These do not accomplish the goal (see point #1).

Chris

Hungarian notation considered harmful

Clear, Consistant Formatting: [...] variables use Hungarian Notation or some other standard

I find Hungarian notation much harder to read than not using it. For example, I find the Unix man page for strcpy which looks like this:
char *strcpy(char *dest, const char *src);
much easier to read than the Windows-style Help which is full of stuff like "LPCSTR lpBuf" and suchlike. The idea which is commonly called "Hungarian Notation" says that a variable name should include the type of the variable as a prefixed abbreviation in front of the name. This leads to stuff like:
byte[] baBuf;
whereas without Hungarian, it might be called:
byte[] message;
which would be much more meaningful.

Especially in object-oriented programming, the type of a variable is the least important piece of information about the variable, and has no place being abbreviated and prefixed to the name. The most important thing about a variable is what the programmer is using the variable for, and that information should be what the name of the variable tells another programmer. If somebody really wants to know the type of a variable, then their editor or IDE should tell them what it is. If it doesn't tell them automatically, then they should look at the variable declaration, which will state exactly what type the variable is. If programmers want the variable name to tell them the type, then what is the point of declarations? And why bother putting a comment near the declaration saying what the variable is for, because people aren't going to read the declaration or comment anyway, because they are just going to look at the Hungarian warts.

The argument that Hungarian notation reduces the possibility of assigning variables of different type to each other is long dead with compilers well capable of throwing errors if any incompatible type assignments are attended. I think that Hungarian notation is dead, or at least should be.

How To Make Software Projects Fail:

Step 1: Hire my boss (God, please hire him away!).
Step 2: Put him in charge of software development.
Step 3: Do nothing as priorities change weekly and deadlines slip away.
Step 4: Do nothing to stem exodus of clued-in employees to less-screwed companies.
Step 5: Force remaining employees to work 15 hour days. Provide subtle reminders that there's a recession out there.
Step 6: Do nothing as even non-clued-in employees flee.
Step 7: Hire a sweatshop in China to crank out code; present this sound like a good idea.

There, that was pretty easy. And, to be honest, everything beyond Step 1 pretty much happens on its own.

Re:How To Make Software Projects Fail:

Ah, the true sign of a clueless boss: A non-AC post about how clueless your boss is, with no fear he'll read /. and find it ;)

Soul of a New Machine (slightly OT)

I'd HIGHLY suggest reading a SoaNM. One of my favorite books (fiction or non).

problem = clueless management/directors/execs etc.

Alot of .com's I've worked at in the past 2-3 years always wanted to "lowball" developers and engineers, while lining the pockets of resource managers, implementation managers, marketing people, etc.

Then a skilled/talented developer and/or engineer wants more money. The employer does nothing to retain them - thus the skilled/talented employee leaves.

Now who maintains the code?

The other problem is bringing in short term consultants for long term projects. The non-technical people who make these executive decisions don't seem to see the feasability of KEEPING their code maintained by the talented/skilled person who BEGAN the development on it.

I know alot of consultants read /. - so I'll probably take a big hit on the karma - but I just was the casualty of another dotcom failure - and this was a seriousl problem.

Another problem is hiring non-technical managers to manage technical people. At my last job we had a manager off of an automobile manufacturing production line quit his job at the auto company to take a job as the manager of a group of Unix admins. This "bumper jockey" had NO CLUE what we did for a living, and treated us like a bunch of unionized UAW slobs, and not like professionals.

How can a non-technical boss effectively manage technical people???

Also - how about all the Ceo, Cio, Cto, eieio - types with their big salaries, catered lunches, etc... Alot of them have NO programming or hands-on technical experience. Hell - I've had the CTO come up to me and tell me that "The Internet was broken" when he knocked the dongle out of the side of his laptop - severing the network connection. And this guy is our Chief Technology Officer???? *lmao*

I'm not saying that only technological people can make technology companies work - but I do feel that managers should take some sort of hands-on classes to learn some basic programming and internet skills so they have SOME SORT OF CLUE about what WE all do for a living!

Death by Engineers

I'm in the alternate situation: Too many of my execs (except, for some reason, the VP of Development) are engineers.

This leads to a whole host of problems:

Many of them tend think they're smarter than people in non-engineering roles.

Pursuant to this, they don't think PR and marketing and sales are "hard" or really even "important".

Again after #1, they're always right when in disagreement with marketing or sales guys.

Most of them haven't developed in a decade+, so now they know just enough to be dangerous -- make micromanaging decisions about detailed subjects things they don't understand well enough, chase unnecessarily after bleeding edge tech, etc.

Fail to understand that not everyone wants to always work 14 hours a day.

Laugh off meetings, so that eventually nobody in the company knows whats going on.

As a result, nobody's heard of us (no marketing budget, no trade shows, no nothing) and nobody's buying our products (engineers tend to make lousy sales guys; despite what they might believe, nobody wants to listen to a 3-hour ridiculously detailed presentation on your product).

There's got to be a happy medium someplace.

Rewrite vs compatability

Joel: Hmm. That reminds me that Microsoft learned the "no rewrite" lesson the hard way.

Obviously, MS biggest problem though is that they don't know when to give up and actually rewrite. For instance, it seems that the windows series of operating systems are all made with the intent of being backwards compatible and reusing core parts back to early DOS systems. Backwards compatability and code reuse is nice and all, but there is a limit to it and a time to give up.

It will however be interesting to see what comes out of the "total rewrite" of IIS.

"never a good idea to do a complete rewrite"

ahem. what was NT for? sometimes, you just have to come to terms with the fact that as tested, bug-fixed and studied as a chunk of code may be, it was developed as part of a misconceived model of either visible functionality or internal architecture or both. DOS and its progeny like win32 were clearly cases of this, and MS weathered a complete rewrite c/o cutler and co. quite happily. the fact that there are examples of disastrous complete rewrites doesn't mean that the examples that worked are meaningless.

Before everyone points at Microsoft .....

True, MS's monopolistic policies notwithstanding:

"Everyone thinks, poor Netscape, they were a victim of MS practices" - yes, they were, and yes, they innovated, but come to think of it, NS4 was crappy software that sucked.

"Poor Real Networks, MS is integrating all that stuff into the OS." - Good riddance ... IMO anything Real makes has never made it out of Beta, and naturally, don't unload the stupid System Tray icon that leaks memory like a sieve, because "you could lose some key features and performance benefits."

We blame Microsoft because their software sucks, and their practices suck ... but come to think of it, the only people to blame is Microsoft's competition, with their heads up their asses who can't put out decent software that works. Windows 95 didn't become the standard because it was great software, Win95 became the standard because OS/2 was marketed improperly, and IBM didn't work hard enough with OEMs. (that's gonna bring on flamage, so go ahead)

Only now, with Linux and Open Source, can WE the users contribute to what we want, not what some guys proposed business model wants. I mean seriously ... I hate Microsoft as much as the next guy, having to put up with their software, but then again, I don't see many other "competitors" really trying ...

ICQ pioneered instant messaging, but give me a break, the things been in beta for years and uses up more memory than most anything.

My note to all burgeoning software companies - Make me something that doesn't suck,and I'll pay for it, don't force me to upgrade every 20 minutes to a more bloated piece of crap that is nothing more than a "portal" for all those neat advertising engines you've snuck in there....and I swear, if I hear someone say "monetize the desktop" .... heh...

Re:Before everyone points at Microsoft .....

Make me something that doesn't suck,and I'll pay for it, don't force me to upgrade every 20 minutes to a more bloated piece of crap...

Unfortunately, if I write software that doesn't suck, doesn't need patches, and does what you want, you'll buy one copy (Netware 3 [novell.com], WinZip [winzip.com], Eudora [emailman.com]) and in 2 years I'll be bankrupt.

If I write software with tons of broken features and requiring constant upgrades for 'compatibility' and security (SAP [sap.com], QuickBooks [quickbooks.com], and Windows 95 [microsoft.com]), I'm guaranteed plenty of repeat customers.

Now if you'll excuse me, I need to go buy a $100 ink cartridge for my $30 printer.

Re:He certanly is into lunch, isn't he?

No no- it IS good capitalism, BECAUSE straight uncut winner-take-all capitalism is not good for society.

What you're seeing there _is_ capitalism- it just happens to be 'laissez-faire'. Under current conditions, those guys are the only ones who survive, because they 'eat the lunch' of everybody else and make sure there's no choice to resort to, by hook or by crook. In strict laissez-faire as it's practiced in the modern world, there is no concept of 'society' at all. It's 100% Union Carbide and there is no such place as Bhopal...

Now, it's important to remember that there are OTHER types of capitalism, but to claim laissez-faire isn't capitalism seems a bit wrong. The trouble here is that you are aware of society and things like consequences to actions, perhaps you are aware of stuff like game theory that proves 'best doesn't always win' and you object to the rules of the game being virtually nonexistent, because you see what happens and you don't like it.

However, to do something about it you'll have to encourage a different sort of capitalism than the laissez-faire one we live with, and until then it will be about 'eating lunch' and to hell with society, customers, or even basic fitness to the task.

...which explains why a lot of people like you are resorting to Linux or otherwise building and maintaining their own cyber 'tools'! One failure mode of this laissez-faire is that there will always be a certain number of people who find it easier to actually do the work themselves, than to struggle with crap every day. 'Eating their lunch' only matters to an accountant- if you're talking about a tool you need, and it's an IMPORTANT tool, then you may be forced to place a much higher value on its quality than a laissez-faire marketplace would ever support.

Code rewrite

A lot of the article is about whether or not you should ever rewrite code.

SMS: Joel, what, in your opinion, is the single greatest development sin a software company can commit?

Joel: Deciding to completely rewrite your product from scratch, on the theory that all your code is messy and bug prone and is bloated and needs to be completely rethought and rebuild from ground zero.

SMS: Uh, what's wrong with that?

Joel: Because it's almost never true. It's not like code rusts if it's not used. The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they've been fixed. There's nothing wrong with it.

Joel blasts rewriting code some more, but doesn't really get into alternatives. Instead he talks about forcing programmers to get with the program, and if they don't, fire them.

Isn't there sometimes a happy medium between completely rewriting the whole codebase and continuing to hack it up? For example, maybe you can identify certain modules that can be isolated and rewritten, then tested rigorously against the old code to make sure they're functionally identical. Or you could separate the old code into a library that just does the computational part of a program, and then write a new GUI around it from scratch.

He takes Netscape as an example, saying the worst mistake they made was to rewrite it from scratch.

I admit that it would have been nice if they released the source code to Netscape 4.x, and not just Mozilla. Even if the code was the most gawd-awful thing in the world, in the years since Mozilla started don't you think we (the open-source community) could have at least fixed some of the more annoying bugs in Netscape?

He's only partly correct about the rewrite thing

With your average application from scratch rewites are probably UnCool for the reasons mentioned in the article.

For other kinds of systems I'm going to have to argue that rewites can be very beneficial.

For systems where the development team has access to a regression test suite and the old (working) code at the same time rewrites are much more easily done. You simply treat the existing code like a prototype. Something that captures all of your requirements, but maybe not using a design that ended up working out (read as: turned into a freaking hairball as time passed!) You work through the old code, understand it, and then build up a new design that works out cleanly in all of the places where the old code was a hack.

When you are all done (or as you are working), you hit it with the test suite. This works out best if your process requires that all of those pesky little bugs found in the old code had to have test cases to reproduce them. (Obvisouly there are limits to this ... the infamous 1 in a million "random" crash, etc.)

Anyway, I think Joel's statement was just a little to broad. He's correct in some cases, but not all. Of course maybe I'm just one of those overly confident coder types ...

why are we listening to this guy?

Hold on, this man worked at Microsoft from 1991 to 1994. He led the Excel team. He led the VB team. This was win16. Excel is great now, but do you remember how much it sucked before office 95? And who the heck used VB for 3.1?

Even better! he wrote the Juno e-mail application. Believe me, this was no fine engineering here. Why does he know better then anyone other Tom, Dick or Harry what makes software project tick?

Not "fron scratch"

Just a correction to a point raised in the interview:

Netscape made the "single worst strategic mistake that any software company can make" by deciding to rewrite their code from scratch.

Netscape didn't rewrite the browser from scratch. Back in April 1998, Communicator 4 was the current version; to get from there to the open-source Mozilla browser, everything that couldn't be distributed (code from other companies, and security code with export restrictions) was stripped out of the source code. What was left was made available as the start of Mozilla. It didn't even compile at first, but Mozilla didn't start from scratch.

Admittedly, the fact that this next-generation browser hardly worked at all for more than three years did keep Netscape from capturing any market share, but the browser had already been commoditized, and the battle had already been lost.

I think that the real browser battle is yet to come -- when the bulletproof and iron-clad Mozilla, carefully fine-tuned to scratch every developer's personal itch, is finally ready sometime next year to take on whatever Microsoft has got. I think that's when the real interesting things will happen -- not just on the technical and marketing fronts, but also on the legal front, as Microsoft finds ways to make sure Mozilla isn't a threat...

Sure-fire way of making a software project fail:

1. Act like a cheezy salesdroid. Promise to implement everything the "customer" (usually some other department of the company) requests and tell them that it will be done in a very short period of time like, say, a month or two. Mutually exclusive features are really good here. Say and do whatever it takes to "sell" them on the project.
2. Talk with the "customer" on a regular basis. Promise to make all changes that they request, especially the ones that would normally be far outside of the scope of the project -- the ones that any sane engineer would insist requires a redesign. Promise that it won't be a problem to make these changes and that it'll only take a couple of weeks at most.
3. Push your developers hard. I mean really hard. They'll have to work 20 hour days for weeks at a stretch in order to meet the design goals and the target release date, after all, and they do work for you, after all, and you did promise the "customer" it would be done on time, after all. When the project gets behind schedule, fire the team lead(s) to provide "motivation" for the rest of the developers and to show everyone that you mean business. They were just getting in the way anyway. It doesn't matter that they had the most knowledge about the project, because we all know that software is easy.
4. When you near completion of the project (assuming your developers haven't bailed out on you already, but hey, the economy sucks right now so they'll be happy to be your bitches), hold another meeting with the "customer". You're almost sure to discover that they didn't really need what you're building that much anyway. Oh, well, at least it was good exercise for your developers! At least, for those that are still around. Hold a meeting with your developers, declare victory, and retreat (um, I mean "advance in the opposite direction").

No, I'm not cynical. Honest.

Program Manager != Programmer

From the interview's lead-in material:

As a Program Manager on the Microsoft Excel team, Joel designed Excel Basic and drove Microsoft's Visual Basic for Applications strategy.

At Microsoft, the job title of Program Manager is given to the people that design the software. They dream it up, write the specs, hold countless meetings, and basically lay the path for the developers to follow. The developers (Software Design Engineer) are tasked with actually programming that software (and thus would be considered "programmers"). Just to round out the roster, the Software Design Engineers in Test (SDET) write the testing suites used by the test teams, and the Software Test Engineers apply those suites to the code following a test plan that they create. In that heirarchy, only the SDE and SDET jobs could be accurately described as "programmers".

Note that this is actually not so cut and dried, wherein SDEs often do design work and test work, and SDETs often do the work of SDTs. PMs don't program, however (well, aside from javascript&html prototypes, anyway).

The point? Calling this Joel an ex-Microsoft programmer is misleading, because he was not. However, the position he held at Microsoft actually lends more credence to his views on design than if he were actually an ex-programmer, as part of the job description of a program manager is doing software design.

(Brief descriptions of all these job titles can be found at Microsoft's college site [microsoft.com].)

This guy is a turd!

Ok, I give him points for pragmatism. I don't think anybody in their right mind will criticize Microsoft for failing to capture market share.

However, I feel slimy for just reading that stuff. Here is what I got:

1. Bugs are fine if they get your product delivered.
2. Load in useless features to drive sales, knowing that your code will suck.
3. Once you have gobs of crap code and a large user base, there will never exist the possibility of re-designing things (eg, WinXP) since it doesn't matter that code sucks (see point 1) and all that counts is revenue.
4. Being efficient is a waste of time. Let the hardware catch up with the crap code.
5. The customer never has valid input anyway.
6. Do it fast and furious, even if January 1900 is broken. Consumers are idiots anyway.

These may be great for sales, but ultimately you will build crap. Garbage in, garbage out. I would rather design good software that was well designed and efficient than vomit up mounds of bloat that will ultimately topple under its own weight.

Software built poorly will never hold up over time. If you look at how little UNIX has had to change over the past 30 years to keep up with "The Internet Age" versus the amount of work done to get XP "working", the future looks bleak for Microsoft. In 20 years, their OS need 25GB of RAM simply to boot up. Of course, this seems not to concern them.

Re:This guy is a turd!

These may be great for sales, but ultimately you will build crap. Garbage in, garbage out. I would rather design good software that was well designed and efficient than vomit up mounds of bloat that will ultimately topple under its own weight.

Everything he said was focused on achieving commercial success. He gave solid examples of times when companies did not do as he suggests and failed commercially. I can't think of too many examples of companies that have succeeded overwhelmingly by doing otherwise.

On the other hand, he did not talk about sculpting perfect software people will use for decades to come. I can think of software that succeeded in this respect, and it didn't do it by following his advice. (TeX comes to mind.)

For example, he talked about bloatware, saying it is a good thing. "Features make users' lives better if they use them, and don't usually hurt if they don't." I disagree with this when talking about "hurt" as "making the software more painful to use" instead of "cutting sales". Extra features introduce more bugs and take away from the time programmers could be fixing other bugs. They shouldn't be added until everything else is perfectly solid and possibly not even then.

Really, I think there is a time for to listen to what this guy has to say and a time to completely ignore it. If you're developing a commercial project, his advice has a certain merit. If you're doing something as a hobby, producing a good piece of software you're proud of is much more important than producing a product before your competitors.

Another interesting read...

From Robert Cringely: Microsoft's C# Language Might Be the Death of Java, but Sun's the One to Blame [pbs.org]

Yes, there are a lot of companies who have been squashed (or, as Joel would say, "Had their lunch eaten") by Microsoft in large part because of Microsoft's money/marketing, but there are also a lot of companies that nose dived into failure because of their own ignorant business and technology decisions.

While Microsoft may not like the costs and annoyance of court cases and DOJ action, it must give them some satisfaction because most of those companies bringing suit against Microsoft are doing so because they think that's their best option. I would argue that for these plantiffs making better products would be a "better option."

A little unrealistic

Joel assumes that crusty code is always filled with knowledge. No, sometimes its filled with crap. More code often means more bugs, not less.

I agree with the spririt of what he is saying - often the "rewrite" is an ego thing - one programmer wanting to write his code instead of reading someone else's, but there is no doubt that most serious professional programmers have looked at code that simply needs to be thrown away.

not rewriting is why ME makes me nau

It's supposed to be a simple function to display a window or something, but for some reason it takes up two pages and has all these ugly little hairs and stuff on it and nobody knows why. OK. I'll tell you why. Those are bug fixes. One of them fixes that bug that Jill had when she tried to install the thing on a computer that didn't have Internet Explorer. Another one fixes a bug that occurs in low memory conditions. Another one fixes some bug that occurred when the file is on a floppy disk and the user yanks out the diskette in the middle. That LoadLibrary call is sure ugly but it makes the code work on old versions of Windows 95. When you throw that function away and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work.

Now, maybe I'm just ignorant because I've never really developed anything with the 640k barrier or had to haggle with XMS and EMS and other whatnot, but it seems to me that those bugfixes are needed because there's something else fundamentaly wrong with the code. IMO, sometimes you just have to rewrite, because your code is just fundamentally wrong.

Take DOS, for example. Microsoft added Windows. Then they made a bootloader, added real multitasking, and called it Windows 95, which wasn't that bad. Then they made some fixes, and called it Windows 98, which supported newer hardware, but was more unstable. Then only God and the developers at Microsoft know what happened to Windows ME, but that's when the bugfixes started causing bugs themselves. I mean, plugging in a USB printer shouldn't freeze the entire system!

Microsoft knew that they had a fundamentally wrong approach to an OS, so they wrote NT, 2K, and are new phasing out ME in favor of XP. XP replaces ME because ME is crap. However, this dude doesn't seem to realize that his own company isn't following is "wisdom."

Maybe I'm just cynnical, but I wonder if there is some kind of ulterior motive here.

He missed one

He missed expecting developers to work 9+ hour days as standard practice. A good book is "Debugging the Development Process". The author also worked at Microsoft, he was a project manager for a couple of different projects that were missing deadlines. He said often they were working 12 hour days all the time. When he started making people go home and also managed the to-do-list better the project would stabalize.

MS successfulness != code quality

According to Joel, the single greatest development sin a software company can commit is "deciding to completely rewrite your product from scratch, on the theory that all your code is messy and bug prone and is bloated...A programmer will whine about a function that he thinks is messy. It's supposed to be a simple function to display a window or something, but for some reason it takes up two pages and has all these ugly little hairs and stuff on it and nobody knows why. OK. I'll tell you why. Those are bug fixes. One of them fixes that bug that Jill had when she tried to install the thing on a computer that didn't have Internet Explorer. Another one fixes a bug that occurs in low memory conditions. Another one fixes some bug that occurred when the file is on a floppy disk and the user yanks out the diskette in the middle. That LoadLibrary call is sure ugly but it makes the code work on old versions of Windows 95. When you throw that function away and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work."

My God. So this is what Microsoft code looks like? It's a miracle it can be maintained at all. This sounds like sloppy coding by trial-and-error, at its worst. Code filled with "ugly little hairs and stuff" that "nobody knows why" is almost a guaranteed recipe for buggy, unstable code. If all these "bug fixes" were properly commented to begin with there would be no argument as to why they should be kept. Thank God for open source, where programmers are _proud_ to show off their code (well, a lot of them, anyway).

I would attribute the successfulness of Microsoft, and the failure of others, to factors other than the quality of its code.

Not relevent to free software movement

This article is really about the economics of rewritting software.

The fundamental difference between free software and commercial software is that free software is about the product, commercial software places a higher value on the profit than the product.

The classic engineering design method is to build something, break it, build it again break it again, etc etc.

Ive never heard of good engineering design comming from build it, break it, fix just the bit that failed, break it, fix just the bit that failed, etc

The whole program is one product, patches dont always fit in nicely with the overall program flow, thats when a program becomes ugly.

Ugly code is more costly to free software because it stops people wanting to get involved, if commercial software is ugly they just pay them more money or something, managment doesnt care how much programmers like readign the code, they just care that it works and its on time/budget

Thinking "simply"...

[1st and foremost: I use and support Microsoft products. The software and OS's work "OK". No further explanation or flaming is necessary.]

Everything this guy says tells programmers to consider the bottom line, the almighty dollar. This attitude works in other industries, but will eventually bite them in the ass. (Automotive anyone?)
He's actually giving us the directions on how to beating MS. So, if you are producing such and are in this to make a fortune today instead of tomorrow... take notes they will be invaluable... for the near future.
However, we all know this is the worst advice for those of us who use and program open source software. We want simple code. We want it to do just the basics. If it's too basic for you, here's the code, feel free to add to it.
Remember the automotive industry? Japan (and Germany) started out with simple basic cars and trucks. And the typical American car buyer? "They're so small, so plain and slow." Hmmm, now these little 4-cylinders are blowing the doors off of the bigger American cars. Because each time they built their cars, they started out simple and refined each part before they added on.
MS is winning today, but soon people will like their programs and OS's like they demnd their cars now, reliable and economical. It will happen, but how long will it take?

To succeed in commercial software...

Let me tell you a bit about the context of everything that I write at Joel On Software [joelonsoftware.com]. Everything I say should only be taken as advice if your goal is to write (a) commercial (b) software (c) for lots of people that (d) succeeds, or at least has a pretty good chance of it. I run a tiny software company, Fog Creek Software [fogcreek.com], in a niche where Microsoft has (at least) two competing products. I'm not any happier about Microsoft's dominance than anyone else. I don't own Microsoft stock. On the other hand, I try to be as rational, logical, and non-religious about every decision I make because it's the only chance I have of succeeding.

Yes, it's true. If you make a major mistake, you get killed, often by Microsoft. Some people think it's a pretty sad state. I just think it's capitalism and evolution. Dodo birds are extinct, and so is Visicalc. I don't want to be extinct, so I try to learn from the mistakes of the companies that have tried to go up against Microsoft. It's easy for me because I was inside and I know something about the way R&D worked at Microsoft. I've tried to share many of these lessons on my site.

To succeed in commercial software you have to get beyond being shrill and angry about Microsoft. You have to be cool headed and smart and study the past and make the right decisions for your company, not the right decisions for some arbitrary sense of aesthetics (although of course I am as big a fan of clean documented code as anyone.)

Production code is not so pretty. Open source code is not so pretty. No real code is all that pretty. It takes time to study it, understand it, and read it, to understand how it got the way it is. The more widely the code is used, the more true that is. For Fog Creek's latest software product, CityDesk [fogcreek.com], we stayed up one night tracking down a bug that only happened on Chinese Windows, where asc(chr(x)) turned out not to be equal to x, an assumption we had been making. How many of you ever thought about getting your code to work on Chinese Windows? No matter how well that piece of code was designed, I'm sorry, I've been programming for 20 years and I never realized that asc(chr(x)) was not always equal to x on some platforms, and I designed it wrong, and until someone tried it on Chinese Windows, I never would have known. Now the code uses byte arrays instead of strings and doesn't have that problem. There's a nice comment in the code saying "use byte arrays instead of strings because of MBCS versions of Windows." The code now works perfectly, but the byte arrays are a little bit uglier than strings. If ten years from now somebody rewrites CityDesk from scratch, I'll guarantee you that 95% of the Windows programmers working today would make the same mistake again, and stay up all night again.

If a piece of your code is ugly and doesn't work, by all means, rewrite that piece. If it's ugly and works perfectly, you're wasting valuable hours rewriting it, time that could be spent doing something that will gain you market share. If you really have an undecipherable mess of spaghetti, 9 times out of 10 you're just being lazy about deciphering it because it seems like more fun to create it from scratch, but it's the ultimate in arrogance to think that your newly created from-scratch version is going to be all that great.

Re:To succeed in commercial software...

People should try putting some damn comments in.

Early on in my programming career, I was looking at such a piece of code - something which was simple when first written (in C) 5 years previously, but had so many mods, the function was pushing 1000 lines (ugh). Some guy had put in this change, commented it as "for performance improvement", then commented it out with an extra comment explaining that it was taken out because it didn't work. At the time I thought "why is this guy making himself look like a dick?". Later I got it - he'd left it like that just to stop some other poor bastard making the same mistake (it was a calculation thing which we were always pushed to improve performance).

If you do something wierd or clever, for f*ck's sake put a comment in (do as I say, nbot as I do).

Joel is almost completely and totally wrong...

He is correct that a rewrite is expensive, and can (and usually does) take a long time.

However, the mistake is not in doing the rewrite, but in not managing the process well.

The number one reason for doing a rewrite is for a cleaner, more stable architecture for writing new features against. The need for a new architecture is discovered in the process of adding new code to the old, and discovering issues that were not adequately addressed in the older version, or in learning better methodologies, or in the existence of better tools and programming processes.

Programs that have been improved by a total rewrite...

Windows NT/XP over DOS (and DOS windows)
Excel over Multimate
Word For Windows over Word for DOS.
Adobe Indesign over PageMaker
Quake over Doom
Quake II over Quake

That is off the top of my head...

ALL real successful software was origninally generated by extremely small teams of EXCELLENT *STAR* quality programmers. (There are not very many of them. If you don't believe that programming is a talent industry, you don't really understand what it takes to make successful commercial software).

The only real other option is unlimited resources (time and money) and it seems that where this exists is at Microsoft, and in some open-source projects.

The biggest problem comes from management believing that random team of programmers can create a new platform from scratch and that it can be done in a schedule that permits dropping the old code base.

ID does it by continuing to build new platforms with very small extremely talented teams.

ADOBE and Microsoft did it, with lots of time energy and effort, with parallel development against the old code base.

But this does not mean that it shouldn't be done. Those that do not rewrite eventually lose, because they are not able to respond to the market on the old code base, and are not able to make the kind of advances that a required by the customer base to upgrade if they use thier product already, or to switch or begin using their product if they hadn't already been convinced.

Managers are going to be disserved in the long term by reading Joels thoughts on the process, and ultimately the companies they work for will be eaten for lunch by new competitors that are not burdened by legacy code, but also really understand well the problem space they are trying to solve.

Oh the religion, oh the pain

Like Joel, I have been programming for 20 years, so I'm certainly not trolling just because what I have to say isn't the in thing with the core of the Slashdot audience.

I read Joel's interview yesterday, before it was mentioned here. Good interview, I thought, he makes lots of good points. But the debate about it here has nothing whatsoever to do with what was said there. Many of the comments key off of the word "Microsoft" and so immediately assume that the interview is crap and has something to do with justifying Microsoft's monopoly position (are these people really bots?).

Most of the the comments, though, are taking little bits of advice and twisting them around into mini-lectures about commenting style or programming issues, or they're simply being used as jumping off points for the poster's own spouting. Let me make this perfectly clear:

These people are not professional programmers.

Anyone who has been through the wringer of commercial software development, and not just a few classes and some tiny open source projects, wouldn't be so religious about such trivialities. Real software development is different. It is not a battle between the Evil Bad Commenters and the Perfection of Beautiful Computer Science (or more correctly What My Professor Said in Class Last Semester). That's not how it works at all. All programmers know about commenting, about indentation style, and so on. There's more to developing commercial products, though: deadlines, missed features, last minute requests from the client, strict requirements for supported platforms, and so on. In this kind of environment, commenting style is a very minor issue (not to say it isn't important, but ranting about it is like ranting to an experienced guitarist about your pet music theories--when you barely know how to play guitar at all). A good way of spotting such people is to ask them what they think of "goto." Odds are you'll get all sorts of vitriol about the evils of goto and the benefits of structured programming and how you should never, never, ever, even if your life depended on it use a goto. An experience programmer would shrug and say "sometimes they are useful, sometimes not."

My advice: Learn, practice, work on projects, and over the years you'll become a pro. A college student without significant software engineering experience is not in a position to rant about how commercial development doesn't fit his ideals. The true sign of experienced developers is that they've been through it all and have enough experience that they don't feel the need to rant every chance they get--or at all.

Death March

Ed Yourdon wrote a book a couple of years entitled "Death March: The Complete Software Developer's Guide to Surviving 'Mission Impossible' Projects [amazon.com]".

Whenever I hear of a software project failing, I think of this book because it explains in gory details what happens when software is treated like fast food instead of architecture.

When Joel Spolsky gripes about "re-writing" as the cause to failure, he's both right and wrong at the same time. Rewrites don't kill projects, MISMANGED rewrites kill projects.

There are some other points that raise my suspicious about Spolsky's training and experience. Since the 90's, there has been a big effort in the industry to develop large scale products with some semblance of reuse. Hence, one of the determinat reasons for the lurch into object oriented program.

Spolsky descriptions sound to me like he's still thinking of code, and of failed projects that were lacked modularity. Nor did he give much attention to other major factors such as FEATURE CREEP, where a small system becomes spagetti over years and years of maintenance. Same with scalability, challenges definately occured in the past decade or so with the massive changes in processors, operating systems and their associated APIs/internals.

But again, it all gets down to one's approach. If you treat software development like you're flipping burgers for the lunch crowd, then you're going to have to deal with the indigestion that comes along with building a house sloppily.