Why Buggy Software Gets Shipped 422
astonishedelf writes to mention an article in the Guardian about the hard reality of why buggy code is sold on retail shelves. From the article: "The world's six billion people can be divided into two groups: group one, who know why every good software company ships products with known bugs; and group two, who don't. Those in group 1 tend to forget what life was like before our youthful optimism was spoiled by reality. Sometimes we encounter a person in group two, a new hire on the team or a customer, who is shocked that any software company would ship a product before every last bug is fixed. Every time Microsoft releases a version of Windows, stories are written about how the open bug count is a five-digit number. People in group two find that interesting. But if you are a software developer, you need to get into group one, where I am."
Windows Software Shop :-) (Score:5, Insightful)
Anyway, I do agree with the author for the most part (its all pretty 101 risk assessment stuff). It is inevitable that software will have bugs in it (especially commercial software shipped to a schedule). This is not really news tho'.
What I would like to see is some vendor honesty. How about making a list of known bugs available to your customer prior to purchase? (I know, I know, fairly warning a customer is madness, etc etc).
Welcome to Group One (Score:1, Insightful)
Theoretically, there is no language that is more or less prone to bugs than any other language as understood in Turing Completeness [wikipedia.org]. Without delving too much into this, it simply states that all languages emulate a Turing machine to some degree and therefore should be capable of everything a Turing machine is capable of (although I don't think this says anything about time/space efficiency). One language may be better supported than another or have more libraries but we are going to assume these issues to be irrelevent to our discussion on applications--let us look as all applications being written in the same low level language that your computer directly understands. I don't want to compare architectures either, let us assume they are equivalently prone to bugs and are basic Turing machines.
If we think about a binary executable program (machine language consisting of ones & zeros) then we must recognize that the program's memory space has many many states. Open up your favorite word editor. Type in a sentence you're thinking about. Highlight part of it and bold it. Highlight a different part and hit escape seven times. Do you think that this scenario was tested?
My point is that it is an impossible herculean task for the developers to test any application in every state. They can come close and smart software design can leave an easier job for the testers but it will never be completely tested.
I would define the term bug as "undesired behavior" and I suggest they be thought of in this manner. I will also make the statement that no piece of software can be garunteed to be free of undesired behavior due primarily to the above analysis of them being amazingly complex machines with a large state space. To take a stab at it mathematically, this browser (Firefox) is operating with a 48,604 Kb memory footprint (I have many tabs open). That's 49770496 bytes or 398163968 bits. Each bit can be on or off meaning that for the amount of memory my browser occupies now, there are 2^398163968 different possible states for any similar sized application running. Now, to add even more complexity, that state space adjusts according to what the application requires for memory.
As our applications become more bloated, the situation only worsens. That is why development phases are either getting longer or requiring larger teams from the beginning of the project (mythical man month noted). At what point does the testing phase end? Hopefully never. Hopefully your software that you acquire is supported until the end of time
If you're wondering how companies can ship software with bugs, you should be wondering how is it possible not to ship software with bugs. You should also understand that there are rigorous standards and practices implemented along the way to prevent the most devestating bugs from escaping. If the company making the software has a history of failing in extinguishing the most glaring of errors, simply stop purchasing their software or demand the support you need.
And lo, there was much rebuking (Score:3, Insightful)
And do we really need that much whitespace [linuxvirus.net] on a news page? I know about that whole '10 words per line' usability mantra, but it looks fucking ridiculous. Why can't all the other website owners just think exactly like me?
Wow, look at all that rebuking. Do I win Slashdot?
(IAJAFSS (I Am Just Another Fucking Smartass Student))
A separate question (Score:5, Insightful)
The idea of QC/testing/beta/whatever the heck you want to call it is that you get as many bugs as you can fix while accepting the ones that will remain behind. That's absolutely correct. However, there are companies - like Microsoft - that are notorious for either being sloppy and not getting bugs they should have, or just straight up not caring at all and rushing a product to market that legitimately shouldn't be there.
The argument can even be extended to good coding practices, like worrying about security fron start to finish rather than after you've entered beta (another well known Microsoft flaw, though they're getting better at it). That reduces the number of bugs to begin with, which in turn gives a better product.
Buggy code is sold because it is demanded (Score:2, Insightful)
In my experience, here's the reason (Score:3, Insightful)
Managers.
Specifically, managers who don't know enough about the project (whatever it may be) and make unreasonable promises to their superiors about shipping dates. It seems that the way most businesses are set up reward managers who complete projects on time or early, rather than the quality of the product. As a result, managers tend to rush development teams through their tasks and the end result is a buggy release. And a manager with a bonus check.
If software shops could change their focus to creating a better release product but with a flexible time schedule...say for instance, rewarding managers for fewer bug reports and service calls rather than completion dates, you'd have an entirely different picture.
Nature of Competition (Score:5, Insightful)
Regardless of the nature of the software development team, the nature of competition remains the same.
Stagnation is costly - delaying a product launch drives people to the alternatives (both due to the alternatives updating faster, and due to the lack of progress seen by the outside world).
Of course, buggy software is costly in terms of reputation, as well, so the end question becomes at what point will the delaying of the release cost us more market share then the remaining bugs will.
Unfortunate from a purists eyes, but it's just the way things go.
bugs, so what? (Score:2, Insightful)
So what's the big deal?
I understand shipping something like bad tires that will eventually kill people should not be done, but anything that does not cause harm or major finantial distress should just be dealt with during the normal lifetime of a product.
I am an embedded systems developer. We take care of the functionality problems before shipping and work on the corner cases as we move along.
There is no way a group of people, doesn't matter how large, can think on every possible problem that can occur.
Show me one thing that's man made and that's perfect and I will eat my shoes.
-later!
Real reason (Score:5, Insightful)
In those cases where it does; e.g. medical/aviation software, usually embedded people take the time. If aviation software designers cut the same corners (w.r.t. bugs vs. features) that office software designers do, planes would fall out of the air and people would die. So they write well engineered software, in well engineered, fault tolerant languages (lika Ada). (Yes, yes, Ariane, but thats the exception that proves the rule)
The real reason buggy software is shipped, is because buggy software is accepted by the market, and people will keep buying it, and continue to roll their eyes when it crashes, because they're completely inured to it, and many of them have reached the conclusion that its literally impossible to write robust, stable software.
It's not, but the profit margins are narrow, and no-one seems to mind (or rather they mind, but keep forking over their money anyway). So no-one bothers to.
Face if folks, we're enablers.
Nice excuse, but not good enough (Score:1, Insightful)
However, for mission critical software such as medical devices that deals with raditions output or heck even slot machines, bugs are unacceptable.
A good example is the Therac 25 where due to bugs, it actually injured people. http://www.ganssle.com/articles/disaster.htm [ganssle.com]
I hope the poster of this article or anyone who is in group 1 will never work on any mission critical software.
Re:Welcome to Group One (Score:5, Insightful)
because we want it NOW! (Score:5, Insightful)
Product ships late because of bug fixes. Why is it taking so long.
Product ships on time with bugs. Why didn't you fix the bugs before shipping.
You just can't win
Re:A separate question (Score:3, Insightful)
Anyway, yes, most people don't care about software, but you also have to look at what the application is trying to do and what you need it to do.
1) Rules of thumb: The last 50% of a project is spent taking it from 95% to 100% (or really 99% as the case may be...)
2) Is the software used for medical purposes? Financial? Downside to those barfing are HUGE. Or is it a game that is meant as a fun diversion? All of those make a difference in people's tolerance for bugs/undesired effects/undocumented features
The consumers have spoken with their wallets. So far they have said that low cost up front (PC) is worth more than a more stable, polished platform (Mac). Less true now, thouh, XP isn't that bad. I use PC for work, Mac for home, Linux/Unix rarely.
-A
Re:Windows Software Shop :-) (Score:2, Insightful)
It is the job of the engineer (whether for bridges or software) to make sure that the mistakes that ARE made are not the critical ones - the ones that make the bridge collapse or the software crash.
Until the human race as a whole attains God-hood (whatever your diety, this is unlikely), this is a fact of life.
My experience (Score:4, Insightful)
Now, as to why bugs don't get quashed quickly:
I see each of these every day!
Re:Windows Software Shop :-) (Score:5, Insightful)
I didn't say all software will have major bugs that lead inevitabley to the collapse of the software. Just that all software will have bugs.
All bridges have defects too you know - the suspension cables will be slightly uneven, or some features of the fascade will be unsymetrical. Bridge engineers make damn sure there are no structual problems that will lead to collapse (but even they fail sometimes).
I wish software engineers would be more like bridge engineers as well, but the cost of failure (and the cost of fixing in the event of a failure) are so different between bridges & software that its not likely to change.
Yeah how about those examples? (Score:3, Insightful)
Read: we got embraced and extended all to hell. What do to? Blame SQL! That's right, the language itself! It "isn't portable". Also blame users! "People who refuse to use SQL Server can't use Vault."
And here's some typical MS morality for you: "I'd probably even patent this algorithm even though, in principle, I believe software patents are fundamentally evil."
I don't expect bug-free software of any real complexity to be shipped often. But the examples are both interoperability problems, and not actual bugs. Looks like an excuse to marginalize the non-windows crowd. "...only affects users on non-Windows platforms, a rather small percentage of our user base."
The numbers are too big (Score:3, Insightful)
The problem is with a number that large, no matter how small the proportion is to code size, the backlash would be huge. No potential customer could hear that number and then actually want to buy a copy. I believe they should disclose as much information as possible. But from their perspective no amount of marketing could make up for the negative impact of disclosure.
Foolishness (Score:5, Insightful)
Normally, those bugs have low Severity or Frequency (or both). Sometimes they have catastrophic severity.
Did you know that the twin towers were built to withstand a direct impact from a 707? [punjabilok.com]
Bugs are a fact of life. They follow from the mantra 'nothing is perfect.'
Re:Windows Software Shop :-) (Score:5, Insightful)
There are several reasons not to do this. Not the least of which is simply that most customers don't want to know what's going on inside the application. They want to click "Button A" and have "Box X" pop up and start flashing - or whatever. They don't know that 300 lines of code went into making that happen and they're fine with not knowing that. They certainly don't want to be forced to actually understand what each of those 300 lines of code are doing. Try to explain it to them and watch their eyes glaze over in a hurry. You can try to simplify the explanation for them, but it'll never be enough because they can't grasp the problem without understanding what goes in to making it work; a necessity in understanding why it doesn't.
The point is that when you get into listing bugs, you have a number of caveats. First of all, in their eyes, you're telling them that the application you're selling them is broken. Sure, those bugs may never actually affect them. They may even be in parts of the code that aren't even used yet because it's part of a feature not currectly activated. But hey, you're selling them a broken program. Remember the Pentium floating point fiasco Intel had to suffer through? What was wrong with their processor? Nothing that isn't wrong with anything ever manufaturered: minor defects that rarely or never manifest under normal operating conditions. What happened when the news got out about the floating point bug that didn't affect more than a hundreth of a percent of buyers - if that? The 'public' went ballistic about it and Intel was eventually forced to do a recall of CPUs that were perfectly functional. Nevermind the fact that AMD and Intel routinely publish a long (and growing) list of 'errata' for their processors that are chock full of bugs. The moment the public got news of a single errata (which, again, didn't even affect them), they demanded replacement "working" products. Are you, as a company, going to expose yourself to that kind of liability?
Basically, the customer doesn't understand the inner workings of your application, doesn't want to understand the inner workings of your application, and if you try to make them understand anyway, bad things will likely happen to you. People can handle unexpected behaviors (bugs) when they're discovered and a promise is made to make it right (patch, updates, etc). But if/when they get wind that you knew the bug was there, and that there were others? That's a receipe for disaster.
What geeks/coders need to understand is that we make up a vastly small minority of software users. Unless and until a vendor is selling products to, and only to those who are informed, knowledgable, and intelligent about computer code, you will never see a list of known bugs right on the box. If it's there at all, it'll be made as obscure as possible so that your average Joe can't find it. Why? Because average Joe will go nuts regardless of how inconsequential the bug is, or how much it would push back the release of the product to stamp out all the (known) bugs.
Personally, I don't blame developers and vendors one bit for obscuring the known problems. When I run into a bug in something like MySQL or PHP, I can find out about it, whereas slightly-above-average Joe (beginner PHP page creator) would have a heck of a time. I find that to be ideal, as I can find the information I need and Joe's not taking up the vendor/developer's valuable time whining about 10,000 bugs he'll never see in his life.
Re:Real reason (Score:1, Insightful)
Re:I Thought It Was Relevant (Score:5, Insightful)
I've probably spent about equal time writing C and writing in higher level languages, and I can promise that I make fewer errors in higher level languages, doing equal tasks. I think anyone with a lot of code under their belts can make similar statements. The closer to the machine a language makes you work, the harder it is to keep higher level details in the back of your head. In a high-level language, you're much less likely to make a low-level error (and any you make will almost certainly be caught by a warnings mode on the compiler, and this leaves you to keep more of your neurons working on, for instance, keeping your database and its wrapper classes working together correctly -- a task that is, perhaps, a simple afternoon's work in Python, Perl, Ruby etc. two days in C# or Java, and a week of hair-pulling in C... and well... I doubt such a thing has ever been done in assembler.
Anyway, drop the semantic B.S. this is a debate about practicalities.
Re:Windows Software Shop :-) (Score:3, Insightful)
That whole article sounds more like "We, in group one, are the cool kids. All the cool kids accept bugs. Why aren't you a cool kid?" It is loaded with the underlying tone of "I'm smarter than you because I accept bugs as normal"
After the condescending beginning, which assumes that every single person in the world has an opinion on software bugs, the loser then goes on to make excuses.
"Bugs are inevitable, so why bother fixing them?"
The idea of fixing bugs is NOT a binary one: Fix all or fix none.
There should be a point where an effort is made to fix as many bugs as possible, or at least the ones that cause major problems. Minor bugs can be acceptable, but the current trend is for show stopper bugs to be left in for whatever reasons.
Then there's the excuse of "fixing a bug may introduce new ones!" TEH NOES!
"It might not work so I won't even try." Seriously, if I had this kind of attitude where I work, I'd be looking for a new job pretty quick.
The entire thing just sounds like a condescending pile of excuses from a lazy, incompetent developer. Maybe he's trying to get a job writing technobabble articles since he probably won't be a developer much longer.
Re:Windows Software Shop :-) (Score:4, Insightful)
Nonsense. Every major open source project has a complete list of bugs for all versions of their product for all to see, usually right on their Web site, which you can read prior to downloading. For instance, if you click on the Release Notes for Firefox, linked to from the front page [getfirefox.com], you'll see not only a list of known issues in the current production release, but there's a link to their Bugzilla database so that you can read bugs that have recently been filed against it.
If people would really not buy a software product if they knew about all of it's bugs, then why would they download and install any open source applications? Because they don't have pay for them? Just because you don't pay for a piece of software, that doesn't mean it costs the user $0 to install and use.
Re:Windows Software Shop :-) (Score:5, Insightful)
"All the reasons are tied up in one truth: every time you fix a bug, you risk introducing another. Don't we all start out with the belief that software only gets better as we work on it?"
Holy shit! I can't imagine being on a team with that attitude? Do you people write tests? There are dependable ways of insuring that changes don't re-introduce old bugs, and if you can't fix one thing without causing a seemingly unrelated problem, you're working with some pretty smelly code.
There's definitely truth in the statement that the need to use software and the need to eliminate bugs can be at odds with each other once the bugcount is low enough that the product is usable. But... isn't that obvious? How many open source projects out there (without version numbers approaching a known irrational number that is...) can really brag of being bug free?
BUT this article is a good reminder of why I'm glad the boxed-software days are coming to a close. Software should either be written with a client present or by people who intend on using the product themselves. Trying to come up with a featureset for some vague "target market" is a horrbile way to write software. Software is unique because it is both engineering and design at every phase of its creation... and without set feature requirements and feedback, there's really no way to know if what you're making is well engineered or not. Anyway, don't work in a closedsource / many customers environment... it's bad for the brain.
Re:I Thought It Was Relevant (Score:4, Insightful)
Why do you dismiss a complaint which speaks to the very heart of the problem? A large class of bugs simply would not exist were a different language used. This is not pie in the sky stuff; it's a real phenomenon.
If one language is less error-prone than another, then an application written in that language will have less bugs.
If an error-prone language is being used to write software, then this surely has to be a reason why buggy software gets shipped. Why are you dismissing people who complain about error-prone languages?
Re:Windows Software Shop :-) (Score:2, Insightful)
Re:Real reason (Score:1, Insightful)
There is nothing wrong with this. I don't buy bullet proof armored windows for my house and car. I could conceive of senarios where I would like them, but it's not cost effective for me to do it. How many of you are guilty of buying the cheap plastic version of item X? Don't tell me you won't accept bugs at the right price point.
Re:Windows Software Shop :-) (Score:2, Insightful)
You fail to differentiate between known and unknown bugs. You can't fix a bug you don't know about.
My general rules/laws/axioms about bugs:
Some bugs are worse than others.
Fixing bugs is good.
Fixing bugs may introduce unknown bugs.
Introducing bugs is bad.
There is always at least 1 unknown bug.
Unknown bugs are found over time.
All bugs cannot be found in finite time.
Uknown bugs may be worse than known bugs.
If your goal is to ship with zero bugs, you never ship. If your goal is to fix every known bug before you ship, you risk shipping a product with serious unknown bugs.
Re:Real reason (Score:3, Insightful)
The problem is, computers are so useful, that customers would (for now) rather put up with bugs than do without. Just as in the early days of the automobile, people would put up with the danger of having the hand-crank break your arm if the car backfired while you started it. Cars were new and had big advantages over horses and mules. Nowadays, however, if you tried to sell a car that was likely to break your arm (or even required hand cranking) you'd go out of business in a heartbeat.
Face it, computers are still in their Model-T era.
No software is perfect or ever will be (Score:1, Insightful)
If you think your software is bug free then either:
a) You are clueless, hopelessly naive and inexperienced.
You may not even know what a bug is. Bugs range from cosmetic bugs such as mispellings in text labels, or the wrong color for a panel, to a bug that causes data corruption or system failure - but they are all bugs - including not meeting the user's expectation of how something should work.
Moreover, almost all software uses third party libs or application modules - ranging from the operating system (and all of its modules), to system libs to language libs, to special purpose libs (such as XML parsers, etc.) that you have little to no control over and certainly don't have the time to test or examine. You just take it on faith that these do not have bugs - but they do. If they are part of your application or you have not found and worked around these bugs, then your app has a bug - and if you think every OS and third party lib is bug free then I have to wonder what planet, nay what universe you are from.
or
b) You (or a designated party) have not tested your software properly.
Most developers are clueless when it comes to testing, and even if they had a clue (like me - I am a developer, but I spent almost 8 years professionally testing mission critical apps, including medical software) the developer is not the best person to test their own software - that job is best done by a professional and experienced tester who has not written the software - preferably someone who has not tested your software before.
2) There is no way to completely test any software application - no matter how simple - you can't even approach it, or even *begin* to approach complete testing of software. Even automation can not test every possible data input with every possible logic path. There just isn't enough time:
Testing a very simple program which just adds two 32 bit integers can take years - assuming you covered just the possible data inputs, and assuming you limited the input to integer numbers, not to mention alpha chars and floats - do the math (at 1 billion inputs per second [an extremely optimistic assumption], it would take over 500 years to test all possible inputs of adding two 32 bit integers).
That is just the possible integer inputs, not every possible logic path for every possible data input - and that is about as trivial a program as you could possibly make.
Most testing concentrates around previously known conditions which caused previous bugs (to make sure they don't resurface) and around other inputs/conditions/scenarios which are most likely to find a bug. The art of Software QA/testing is not to prove that there are no bugs, but to find the bugs that we know are there in all software - with priority given to "serious" bugs (what is "serious" depends on the context).
3) The severity or risk of any given bug is highly subjective and contextual. In some contexts a program that crashes routinely is preferable to a program that gives the wrong answer. In other contexts the reverse is true. In some programs (minesweeper?), either crashing or wrong answers doesn't mean a whole lot except that the user may go on to some other program.
Saying that Windows has thousands of bugs doesn't mean a whole lot until you start categorizing the bugs and analyzing the risk vs. benefit of the bug vs. the OS - and that depends a lot on the intended use.
If you don't want to use software that has bugs, better get out the pencil and paper.
The job of developers, QA staff and project managers is to weigh all of these factors and to do the best job we can given the human resources, time and money we have. No software is ever perfect or ever will be. Deal with it.
Re:Foolishness (Score:3, Insightful)
This is directly analogous to the discussion of bug severity in the article. The engineers decided that the severity / frequency of risk coupled with the cost of further upgrades did not warrant the improvements. Either they underestimated the severity/frequency and should have done further upgrades or they made the right call and we just have to live with the fact that we can't fix every "bug" in every design. If you don't like to think of it as a "bug," think of it as a shortcoming or limitation.
I'm sorry for your loss, but there are still lessons in engineering that can come from it. It does no service to the memories of those lost to ignore those lessons. That this event was caused by people with ill intent rather than natural randomness is utterly irrelevant. The fact is that every system has limits beyond which it will fail.
Re:Windows Software Shop :-) (Score:3, Insightful)
I don't think you've thought this thing through to the real world economic implications.
What you Want vs Ask For; and hopes for the future (Score:3, Insightful)
I share your vision of hope for the future. But I would first like to digress for a moment on your statement before fleshing out how I DO agree with you, too. (BTW it is currently 4 AM so I apologize if this rambles a bit. I've tried to go back and edit it to make it clearer, but I seem to keep making mistakes. That in itself seems apropos to this discussion. <grin>)
In my experience, it's more like: the customers get what they asked for and then find out they did NOT get what they WANTED. The problem is that the customers do not understand software, the environment the software runs in (hardware, political, and legal), what is possible, and what has never been done before. It's more often the case that "they will know it when they see it, but they can't really tell you what THAT is, exactly, before hand." Further, because they do not know what is and is not possible, they don't understand the ramifications of their choices. Lastly, there is a HUGE difference between research and development. It's one thing to code a one-off of something you've done many times before; it's quite another to do something completely new, and get it right the first time. As more and more people become computer literate, and gain first-hand experience on using software, I am cautiously optimistic that this disconnect will diminish over time.
Helmets - As an example, I remember watching some old footage and was amazed to see that professional hockey players did not wear helmets. Now it sure likes ALL players wear them. Why? We learned the added inconveniece and expense was worth it. I've worked on development projects where there was no allowance (i.e. time and money) set aside for anything but the barest amount of testing. Now I am increasingly seeing that built into development schedules, as a matter of course. Granted, in some cases this testing would be analagous to a hockey player wearing only a LEATHER helmet (instead of today's high-impact plastic) but it shows progress and gives me hope for the future. I welcome the day when testing and quality assurance are an integral part of EVERY development effort instead of a rare luxury. The benefit is that libraries of well-documented and thoroughly-tested code will become increasingly available. AND the methodology to USE them PROPERLY, (i.e. SAFELY!)
It's thanks to developers' incessant optimism, I believe, that we have our current problems, and also the seeds for the solution. Please, don't get me wrong on this one. IIRC, it was Tesla who said that "If I had known it was impossible, I wouldn't have done it." We create doftware to do things which have NEVER been done before, ever. Even in the face of statements like: "That'll NEVER work!". The response? "Oh yeah? Hmmmmm. Wait a minute! If I, hmmm, and then... AHA! I think that just might work!" And on nothing more than a hunch, a hope, and a blind ignorance of just how many hours of debugging it will entail, we regularly go off and do something absolutely incredible. I have gained much inspiration from a quote by Mark Twain:
In tandem with this one by Thomas A. Edison:
It's that thrill which drives so much inspiration and pers
Re:Windows Software Shop :-) (Score:2, Insightful)
Risk. RISK. Riiiiiiiisk.
How can you even take umbrage with that?
Any time you change a code base, you RISK introducing a new bug. You cannot, cannot argue with that.
Re:Windows Software Shop :-) (Score:3, Insightful)
You're new to this aren't you.
If you can find a way to PROOVE that a less than trivial fix hasn't broken something else, you'd be too busy sailing the world in a multi million dollar cruiser to post to
If you're working on a word processor or other piece of non-critical software no problem. When billions of dollars or lives are at stake you'll need to mitigate the risk with something other than "if you can't fix one thing without causing a seemingly unrelated problem, you're working with some pretty smelly code". Are you willing to bet your life on your code fix? How about the lives of your loved ones. There's plenty of software in a hospital or on an aircraft that must be that good.
As for tests they can only tackle a very finite number of situations. They're definitely good for finding bugs that have been introduced. They're no use whatsoever for prooving you definitely haven't introduced a new bug.