Like all work, software development can be categorized into three groups
- Problem definition
- Problem solving
- Routine work
Routine work is defined as work where both the problem and solution is known. In problem solving, the problem is known, but not the solution. The hardest group is problem definition, where neither the problem nor the solution is known.
When people are comparing software development with building a house (a ridicules metaphor but one that seems to stick around), they are talking about routine work and to some degree problem solving. Software development in these two categories can benefit from rigid processes control and following some form of software development method.
There have been a few articles about the teams that build software for the space shuttle program and space probes and how well they build software, with close to zero bugs, delivery on time and within scope. It can be useful to look at their techniques, methods and tools as well as the development process. However keep in mind that these groups build software mainly in the problem solving and routine work category. Building software after a stable blueprint is "easy" and as long as you have good developers and follow a proven process it is hard to fail. Not that I don't admire those guys, but I am not surprised that they succeed.
However, the reason software projects often fails spectacularly is because they fall into the problem definition category. That is; no one really knows what the problem is and there is no clear solution. In this case, the target (i.e. realization of the specification) is fluctuating and because of this, it is very hard if not impossible to create a good project plan, allocate resources and estimate the cost.
Projects in the Problem solving and Routine work category will almost always succeed no matter which software development method, techniques and tools are used. In fact, the often debunked waterfall method can be useful for many of these projects.
For projects in the problem definition category, using such methods will be fatal. Instead some form of iterative evolutionary development process must be used to try and narrow the scope, follow the target and at the same time have some form of forward progress. There is no silver bullet as Fred Brooks concluded a long time ago. If you only have one set of tools (method) in your box, like most consulting companies, you will fail sooner or later.
The solution is to realize that each software project is different and to have a box full of tools, techniques, principles that can be applied to different situations and problems. And in this case; "death to methods!" Following a software development method for projects mainly in the Problem definition category is fatal.
