Bug Hunting Open-Source vs. Proprietary Software 244
PreacherTom writes "An analysis comparing the top 50 open-source software projects to proprietary software from over 100 different companies was conducted by Coverity, working in conjunction with the Department of Homeland Security and Stanford University. The study found that no open source project had fewer software defects than proprietary code. In fact, the analysis demonstrated that proprietary code is, on average, more than five times less buggy. On the other hand, the open-source software was found to be of greater average overall quality. Not surprisingly, dissenting opinions already exist, claiming Coverity's scope was inappropriate to their conclusions."
Re:So how did they test the proprietary software? (Score:5, Informative)
I don't know if the closed source statistics are online somewhere, but these are the open source statistics.
http://scan.coverity.com/ [coverity.com]
and if you ask me the "Defect Reports / KLOC" is pretty low, and such software would normally be considered "good" software.
Re:How much is it really true? (Score:3, Informative)
An open source software is tested by a whole lot of people over the world and everyone is free to take the code and test if. On the other hand in case of proprietary software this is not the case and is tested by far less number of individuals.
That sounds rather idealistic... The coverage on OSS varies a lot. Most is not tested much, and the testing is not systematic and analyzed, but ad hoc. And if a bug is found, many just shrug and think of it as buggy software, but don't do more about it. There is a world of difference between the high standards of large projects like the linux kernel, apache and eclipse vs. the thousands of small projects found on freshmeat or just googling for something to scratch your itch.
Misquoting TFA (Score:5, Informative)
TFA says that no open source project is as good as the BEST of proprietary, but it also says that the AVERAGE open source is better than the AVERAGE proprietary.
Actually (Score:2, Informative)
It was not an apples to apples comparison, more like apples to diamonds. Dom't worry, just fix any real problems identified. Many of the bugs found are theoretical, not real. Many others are style questions. the experts will probably never quit arguing about what is 'good' and what is 'bad'.
There are also some very real race conditions, memory leaks, and things like that. A real list by line number would be nice.
That said, what would really matter is a comparison program by program. I don't think my quick and dirty one off is really in the same class as the Kernal, or Firefox,I havn't seen how this diferentiates.
Re:So how did they test? -- badly (Score:3, Informative)
A method known to have flaws. It raises a ton of false positives, things that might "look like" potential bugs but aren't because of the data flow. You have to do a data flow analysis to see if they really are bugs.
For example, not checking for buffer overflows when copying strings, etc, is usually considered a (potential) bug. Certainly it is when dealing with unknown input. However, in a function buried deep behind layers of other code that has already filtered out potentially overrunning strings, it is quite safe. (At least until said code is changed, so it is considered bad practise, but the fact remains that if there's no data path that can cause a problem, it's not a bug.)
Mind, there's bugs and there's bugs. The Space Shuttle software is generally considered to be about the most defect-free software around, but they'll launch with some bugs. Specifically, minor bugs in the display (ie a misspelled word, a column that doesn't quite align) are allowed because they're a much lower risk than going in and changing the software to fix them (and possibly introducing a different bug).
Re:Exactly what constitutes a software bug? (Score:3, Informative)
Security holes. Coverity specializes in programmatic detection of buffer overflows.
Oh, and I forgot some of the other obvious things you can check for: unreachable code, comparisons that always evaluate to true or false, possible uninitialized use of variables, global and/or heap storage of pointers to variables on the stack.... There are a lot of things that are usually unsafe to do and are usually bugs. It is usually too slow to check for this stuff during compilation, as it requires at least some degree of static (and possibly dynamic) function call tree analysis.
Re:just an example of how "buggy" OSS software. (Score:4, Informative)
linux 2.6: 3,315,274 lines of code, 0.138 / 1000 lines of code.
kde: 4,518,450 lines of code, 0.012 bugs / 1000 lines of code.
So far so good! But for contrast, I'll add this stat from TFChart:
Gnome: 31,596 lines of code, 1.931 bugs / 1000 lines of code.
Eeeep!!
(No wonder I prefer KDE
Re:What's a bug? (Score:4, Informative)
I think you're conflating two things. The check was (is?) for $50 or some such. The version number of the software is pi (or e) to whatever number of decimals, where each subsequent release adds a decimal place (becomes a closer approximation to the real thing.)
No, his concept of a bug is a deviation from the specified functionality.
That's the only reasonable definition of a bug in the software.
But what if that functionality is wrong or sucks?
Then that's a bug in the specification or in the requirements. I spent the better part of six months debugging the requirements on a major project once. Part of that was getting mutual agreement from three major customers, part of that was resolving internal inconsistencies in the requirements document, and part of that was a high level design process in parallel, to be sure we had a chance of actually satisfying the requirements.
Of course the end user (especially of off-the-shelf software) generally doesn't differentiate between a bug in the software vs a bug in the specification or requirements. The end user generally never sees the spec, and only has a vague idea of the requirements. (Sometimes worse than vague -- how many people do you know who use a spreadsheet for a database?)
(And to BadAnalogyGuy -- I'm not disagreeing, just amplifying.)
Re:Why is this surprising? (Score:5, Informative)
Says who? QA and testing covers the entire gamut, from formalized unit-testing at every level, to 'throw it at the beta testers and hope nothing breaks'. it's got nothing to do with 'proprietary' (not 'propriety') vs open source.
Where on Earth did you get that? Are you completely oblivious to all the testing methodologies and systems developed by the open source community? Here's a few for you to research: JUnit, Test::Unit, and Selenium.
Again with the generalizations! Commercial software development is, by definition, proprietary, so you don't know how they do it! They might tell you they have a 'strong QA engineering component' (whatever that means) but they could be full of shit!
A bug can be many things (Score:4, Informative)
- This is the typical meaning behind the word "bug".
- That's where your TeX complaint would fall under. It's "by design" that it doesn't have an iconic user interface, but that doesn't mean it's something that shouldn't be addressed ever
- This is actually a result of the bug tracking system that we use. Rather than sending e-mail, which often gets lost, we often track work items as bugs. For example, "Need to turn off switch X on the test server when we get to milestone Y"
To further complicate things, there is a severity and priority attached to every bug. Severity is a measure of the impact the bug has on the customer/end-product. It can range from 1 (Bug crashes system) to 4 (Just a typo). Priority is a measure of the importance of the bug. It ranges from 0 (Bug blocks team from doing any further work, must fix now), to 3 (Trivial bug, fix if there is time). (I don't know why the ranges don't match, BTW, seems silly to me)
As anyone who works on large-scale project probably knows, there are always a wide range of bugs, across all the pri/sev levels. To me, a simple count of all the bugs isn't terribly useful. A project could have a ton of bugs, but most of them being DCRs (which are knowingly going to be postponed till the next release) and/or low pri/sev bugs. Or maybe it's the beginning of the project and they're all known work items. Or a project could have only a few bugs, but with all of them being critical pri/sev ones.
So, whenever I see a report that simply talks about bug count, I take it with a huge grain of salt. If I had to guess (I skimmed the article), it seems like OSS projects have far more bugs, but perhaps lower pri/sev since the product itself has been evaluated as being higher quality. In the end, it's the quality that the customer really cares about.
Re:What's a bug? (Score:2, Informative)
Re:How much is it really true? (Score:4, Informative)
No they are not to the extend of a experienced developt.
going through the code dow not find bugs. Either you do a formal correct approach, that is a walk through or a code inspection then you may find bugs, or you only have the chance to find occasional off by one errors in a loop or array index. Just by looking over code as you say in your n00b appoach you only find suspicious pieces of code.
What now? You change it to be less suspicious? And then? You commit it? So you don't know if somethign elsewhere is breaking now because of your change? Ah
Testing means to DEFINE how individual pieces of code should behave and writing a test case exactly for that. Changing software and fixing bugs means to have tests, lots of tests, not eyeballs.
angel'o'sphere
P.S. that does not mean that formal walk throughs / inspections don't work, they do!! But informal ones are only for educational purpose intersting.
Re:Exactly what constitutes a software bug? (Score:3, Informative)
Somebody please explain to me exactly what kind of software bug can be found by automatic scanning that isn't found by standard debugging and compile-time checks. If a computer can ascertain exactly what the programmer intended to do, why do we need programmers?
Decimal one = 1;
Decimal two = 2;
one.add(two);
System.out.printline(one);
Guess whats printed? Similar errors are made if you use methods on java.lang.String like replace(pattern, replacement, pos).
The simple answer to this is that they can't.
Thats a very uninformed oppion! Tools like http://pmd.sourceforge.net/ [sourceforge.net] have a data base of over 100 "bug patterns" to check your code againt. That does not mean all found points are truely positive, but thy definitely are bad coding practice and my end in a bug later if the code gets changed. There are lots of simialr tools, check IBMs alaphaworks and developerworks e.g.
angel'o'sphere
Re:Misquoting TFA (Score:2, Informative)
Re:So how did they test? -- badly (Score:3, Informative)
A guess is that this is because much of emacs' functionality is implemented in elisp code, which is not part of the core program and so not included in the source line count, whereas most of vim is implemented directly.
Re:Exactly what constitutes a software bug? (Score:2, Informative)
Another paper from UC Santa Barbra and (I think) the Technical Institute of Vienna used static analysis of compiled code (not even source!) to try to determine if a kernel module was malicious. (In this context, "malicious" means that it acts like a rootkit; in other words, if it modifies internal kernel data structures that don't belong to the kernel module itself.) They tested 7 rootkits and the entire driver database that comes with Red Hat. Their tool god a perfect detection rate -- no false positives, no false negatives. (The paper does have some issues -- like how did they pick the rootkits they tested, did they analyze them and then build the tool or the other way around, are there other rootkits they didn't test, how easily would it be to make a kernel module that would pass their test, etc., but it does provide a demonstration of the sort of power that static analysis can give you.)
Re:meaningless, no data, and probably biased (Score:3, Informative)
It should, however, be remembered that coverity does not and cannot find all bugs. It is just a more advanced form of Lint that catches a class of errors that is otherwise annoying to track down; a nice extra tool, not a magic bullet.
Automatic code testing (Score:2, Informative)
Automated tests are often more expensive to write than manual tests, so when should you do automated tests versus manual ones? The answer is not always automated tests. Also typical unit testing is only part of what kind of testing needs to happen.
Using Ring Buffer Logging to Help Find Bugs [visibleworkings.com] was also another good link that Federico had in another entry.
openoffice (Score:3, Informative)
since then many people try to clean it, but its hard and risky to clean a such big app
most projects have a coding style that everyone should follow, and many force you to comply if they want their code to be accepted
OpenOffice is a bad example. (Score:4, Informative)
It's too bad because it actually works kinda okay, but it's a real effort to get your hands dirty with.
Blender is also like that... it seems when a codebase has 'gotten around' it tends to pick up the bad habits of all the hands its been through.
MySQL is a bad state because it's really only developed by MySQL AB -- no one else is contributing to it so they have no reason to make it any more maintainable than it is. PostgreSQL, on the other hand, had the luxury of being the fruit of some academic research projects and was rewritten once or twice, so it's a little more maintainable.