Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×

Firefox Analyzed for Bugs by Software 226

Posted by CowboyNeal from the killing-bugs-dead dept.
eldavojohn writes "In a brief article on CNet, a company named Coverity announced that Firefox is using software to detect flaws in Firefox's source code. Even more interesting is the DHS initiative for Coverity to use this same bug detection software on 40 open source projects." An interesting tidbit from the article: "Most of the 40 programs tested averaged less than one defect per thousand lines of code. The cleanest program was XMMS, a Unix-based multimedia application. It had only six bugs in its 116,899 lines of code, or .51 bugs per thousands lines of code. The buggiest program is the Advanced Maryland Automatic Network Disk Archiver, or AMANDA, a Linux backup application first developed at the University of Maryland. Coverity found 108 bugs in its 88,950 lines of code, or about 1.214 bugs per thousand lines of code." We've covered this before, only now Firefox is actually licensing the Coverity software and using it directly.
This discussion has been archived. No new comments can be posted.

Firefox Analyzed for Bugs by Software More

Firefox Analyzed for Bugs by Software

Comments Filter:

  • Math (Score:5, Informative)

    by Anonymous Coward on Saturday August 12, 2006 @11:50AM (#15894412)
    That's .051 bugs per thousand lines of code for XMMS, an order of magnitude better.

    • Re:Math (Score:5, Informative)

      by It'sYerMam ( 762418 ) <thefishface@@@gmail...com> on Saturday August 12, 2006 @01:56PM (#15894981) Homepage
      Quite possibly because XMMS is practically stagnant. I don't even use it, but Amarok is by far the best audio app I've tried for Linux, quite possibly because the people developing it have some idea of which decade they're in.
      • Amaraok is the best Audio player I've used for Linux, Windows or Mac. My only beef is that the lyrics feature doesn't seem to work anymore. It can't get the lyrics, for any songs.

      • Re:Math (Score:3, Insightful)

        by Bert64 ( 520050 )
        Surely a program having very few bugs is reason for it to be stagnant, very little work needs doing in the way of bugfixing and the program already provides all the features it needs (i personally hate programs which suffer from feature bloat - pointless features added for the sake of it, that usually result in the core program not doing it's original task as well as it used to)
        • In my opinion, and that's all it is, the difference between XMMS and a player like Amarok (which I'd damn well use, but I'm on GNOME and don't want the overheads) is well worthwhile. Not to mention that Amarok is simultaneously more usable and prettier than XMMS. Not that I'm trying to convert anyone, I just find it personally irritating that the one player I find that provides the combination of features and ease of use (and, coincidentally, prettiness) is not really practical, while the most well-known pl
      • Why dis XMMS if you don't use it? Is there any reason to be bitter? For many people, myself included, XMMS is still the best out there. Not everybody feels the need for music libraries.

    • Re:Math (Score:3, Informative)

      by Schraegstrichpunkt ( 931443 )
      That's an interesting datum, since XMMS (well, XMMS2, actually) is in the middle of a re-write because the XMMS code is said to be quite inflexible. And it reportedly has a lot of bugs [debian.org].

  • If this is the same (Score:3, Interesting)

    by Anonymous Coward on Saturday August 12, 2006 @11:53AM (#15894425)
    If this is the same as most automated testing software I've seen, it detects many things which aren't truly bugs as bugs. Accuracy on automated testing tools I've been exposed to is around 40%.

  • I will definitely take another look at Coverity's products, if the Firefox team is finding value in it.

  • this slashdot news is already outdated (Score:5, Informative)

    by msh104 ( 620136 ) on Saturday August 12, 2006 @11:56AM (#15894444)
    if you look at the coverity site ( http://scan.coverity.com/ [coverity.com] ) you will see that there are already multiple projects who have brought there bugs down to zero. samba being on of the earliest.

    • Re:this slashdot news is already outdated (Score:5, Insightful)

      by StrawberryFrog ( 67065 ) on Saturday August 12, 2006 @12:03PM (#15894477) Homepage Journal
      there are already multiple projects who have brought there bugs down to zero.

      You mean "who have brought down the count of their bugs that this tool can detect down to zero." I'm sure they will have other bugs in code and design.

      How does this tool compare to tools that do analysis by introspection on bytecode from languages like C# and Java. I use FxCop [gotdotnet.com] on C# code, and while it is very cool, using it is not newsworthy at all. Does this tool do more? Is is the news that it's used in a high-profile C++ program?

      Integrating tools like this into your build process may be cutting-edge best-practice at present, but give it a while.
      • Yes. Automated testing/code inspection is handy, but a signifigant source of bugs is actually missing logic paths, or combanatorics errors which can't be detected by tools generally.

      • Re:this slashdot news is already outdated (Score:5, Interesting)

        by bcrowell ( 177657 ) on Saturday August 12, 2006 @12:14PM (#15894540) Homepage

        You mean "who have brought down the count of their bugs that this tool can detect down to zero." I'm sure they will have other bugs in code and design.
        Yeah, if they could make a program that would detect all bugs in a program, it would violate Turing's proof that the halting probelm is undecidable. [wikipedia.org]

        From the articles, it sounds like they're basically looking for mistakes that could lead to security flaws, e.g., buffer overflows. If AMANDA is particularly buggy by their metric (detectable bugs per thousand lines of code), it's probably because AMANDA doesn't interface to the web, so the people coding it knew that certain classes of buffer overflow "bugs" wouldn't be a problem, because they wouldn't be exploited through an internet-facing interface. If you went back and ran this program on Unix apps written in C from the 1980's, you'd probably find zillions of bugs, but it wouldn't indicate low quality, it would just mean that the programs weren't written for an internet-facing environment in the year 2006, when the internet has become a battle zone for evil spammers, botnets, etc. If the only way such a bug can show up is for the user to supply carefully tailored input, and the result is simply that the program dumps core, then that's not a bug for a program that isn't facing the modern internet.

        • if they could make a program that would detect all bugs in a program, it would violate Turing's proof

          Unless the program's domain is restricted to context-sensitive languages. In fact, it is impossible for a computer to try to decide anything more general than a context-sensitive language because anything bigger requires the resources of a Turing machine, which has infinite memory. Computers implementable in a finite amount of matter are equivalent to linear bounded automata [wikipedia.org], not Turing machines.

          • So? (Score:2)

            by MarkusQ ( 450076 )

            You say this as if it invalidates his point. Since (as you would obviously agree) no computer is more powerful than a Turing machine, if something is impossible for a Turing machine it is necessarily impossible for a computer as well. If anything, your quibble makes his argument stronger.

            --MarkusQ

            • You say this as if it invalidates his point.

              It does. The halting problem proof applies only to machines with infinite memory. There exists a trivial algorithm to determine whether a program halts when run on a linear bounded automaton: given a capacity of b bits and s states, run b*s*2^b steps and see if the machine has halted. This reduces an undecidable problem to an NP-hard problem. Further optimizations are possible, such as by detecting obvious cycles in the state of the machine and/or by recognizing

              • If I understand you correctly.

                Suppose I have a 1Gb (=2^20) memory and some smallish number of states (say 2^8). Then we would need to run the program for 2^20 * 2^8 * (2^(2^20)) steps. This might take a while - if I were to start it running tonight on a nice teraflops machine (2^30 operations per second) it would be done in only about 2^(2^20) seconds. I'll start it running tonight and let you know when its done.

        • Have you ever heard of local exploits? You don't need to have internet to be vulnerable. Applications should prevent from buffer overflows because there can be local attacts as well. You would not want a local user to crash your public workstation or get root on it do you?
          • The program need to be suid or run by root (on "hostile" input") for a local exploit to be relevant.

            Of course, amanda probably run as root on "hostile" input, so local exploits can be relevant.
      • I kind of assumed that without clarification because it is, you know, common sense.

    • No major piece of software is ever bug free.

      I follow the news:linux.samba [linux.samba] Newsgroup a bit. Various Samba features have been shipped broken in various recent releases.

      CIFSfs? (it is replacing smbfs and some Linux distributions have taken to disabling smbfs in the kernel to force people to switch) Cifsfs was broken in the newest major release. An intermediate release fixed that.
      'Valid Users' used with 'smbpasswd': that was broken in the intermediate release. The next intermediate release will cover that.

      N
    • Thank you for that link, it rather nicely confirms that at the time amanda was inspected last, it was clean.

      --
      Cheers, Gene

  • Interesting... (Score:3, Interesting)

    by porkThreeWays ( 895269 ) on Saturday August 12, 2006 @11:58AM (#15894451)
    I find the AMANDA results interesting because AFAIK it hasn't recieved a code rewrite since the early 90's. I think an interesting study would be the to compare older projects with ones that have been rewritten from the ground up. Comparing the rate of new bugs introduced as opposed to those hidden in legacy code.
    • zmanda [zmanda.com] and other amanda hackers [yahoo.com] have been actively developing AMANDA. While the comparison of bugs in new code and legacy code might be interesting, one wouldn't really see this by just counting projects.
    • Not in the least bit interesting. Amanda is under constant development, and to say that it hasn't been touched since the 90's is an outright lie. You work for Veritas or Arkiea?

      The current stable snapshot is amanda-2.5.0-20060424..tar.gz. And there are, sitting on my hard drive, about 12 versions of the next generation that will lead to a 2.5.1 release in a month or so.

      Please put brain in gear and go check your so-called facts before spouting off in front of nearly a million /. readers and making yoursel

  • Bug/Lines of Code (Score:5, Funny)

    by X43B ( 577258 ) on Saturday August 12, 2006 @12:00PM (#15894460) Journal
    "It had only six bugs in its 116,899 lines of code, or .51 bugs per thousands lines of code."

    Sounds like someone needs to run this debugger on their calculator.

  • Once detected can they also fix the flaws? (Score:3, Funny)

    by Browzer ( 17971 ) on Saturday August 12, 2006 @12:03PM (#15894476)
    Or that job is left for the monkeys banging on the keyboards.

  • Which type of bugs? (Score:3, Informative)

    by Erixxxxx ( 920617 ) on Saturday August 12, 2006 @12:09PM (#15894514)
    One has to wonder if these are coding/language bugs or logical bugs. Finding coding bugs is of course a valuable time saver, but the challenging and usually most costly bugs are of the logical sort, and invariably app specific.

  • "Meh. So much for the 'many eyes' theory" (Score:5, Insightful)

    by rjamestaylor ( 117847 ) <rjamestaylor@gmail.com> on Saturday August 12, 2006 @12:13PM (#15894531) Journal
    Even more interesting is the DHS initiative for Coverity to use this same bug detection software on 40 open source projects.
    Before the F/OSS nay sayers toss out the obligatory (and to be expected) "Meh. So much for the 'many eyes' theory" let's point out that having the ability to run a code checker on source code is only possible to the holders of said source code. So, while absolutely true that a proprietary vendor can run the code checker on their code as well as an open source project, there is a huge difference when it comes to the customer/user of said software: with Open Source the user has the freedom to run such a tool over the source code themselves.


    In this age of SarbOx and risk management there is a real competitive advantage to F/OSS over proprietary code to large companies: audit-ability. In previous roles I've had to attest under HIPPA::Security that proprietary code was "secure" -- how? All I could do was obtain a vendor statement that was as non-commital and burden-shifting as possible. Yet, with a true ability to audit the code my pharmaceutical company depended on it would tilt the balance between similar-featured Closed vs Open source solutions. Especially today.

    Ok, maybe nobody really cares about the 'many eyes' theory anymore. Regardless, the "open the hood" theory still applies, perhaps more than ever.

    • So, while absolutely true that a proprietary vendor can run the code checker on their code as well as an open source project, there is a huge difference when it comes to the customer/user of said software: with Open Source the user has the freedom to run such a tool over the source code themselves.

      Actually, I would argue that it isn't just a freedom, it's a necessity. Having the source open means that wrongdoers can use bug-seeking programs to find exploits (presumably they have already been doing so for

  • Meanwhile... (Score:3, Funny)

    by Skiron ( 735617 ) on Saturday August 12, 2006 @12:14PM (#15894537)
    Coverity segfaulted whilst auditing MS Vista.

  • For those who are interested in Firefox' results (Score:5, Interesting)

    by RebelWebmaster ( 628941 ) on Saturday August 12, 2006 @12:14PM (#15894539)
    Here are some links to show the bugs in the Bugzilla database which were turned up by Coverity.
    Open Coverity Bugs [mozilla.org]
    All Coverity Bugs [mozilla.org]

  • Why AMANDA is buggy (Score:3, Insightful)

    by swordgeek ( 112599 ) on Saturday August 12, 2006 @12:26PM (#15894589) Journal
    AMANDA could easily be the buggiest OSS program in existence, and it would still be OK. The reason? It just has to be less buggy than Netbackup, and more usable than Legato. Luckily for the AMANDA developers, this are very very difficult criteria to miss.
  • Coverity's own site [coverity.com] shows how many defects each product has fixed. the number of outstanding defects on AMANDA is now zero. zdnet reported the fixes back in April. [zdnet.com]

    Those that follow amanda-hackers will know that there was less than a week [yahoo.com] between when coverity released the report on March 6th and it was announced that all bugs were fixed in AMANDA on March 12th.

  • No rsync? (Score:3, Interesting)

    by ortholattice ( 175065 ) on Saturday August 12, 2006 @01:06PM (#15894756)
    Funny selection of programs; I don't see rsync on the list. From the article: DHS wants to reinforce the quality of open-source programs supporting the U.S. infrastructure. So, XMMS (an MP3 player) is more important to the U.S. infrastructure than rsync?

  • In defense of amanda (Score:3, Informative)

    by Almost-Retired ( 637760 ) on Saturday August 12, 2006 @01:06PM (#15894757) Homepage
    I'm somewhat surprised to see amanda being badmouthed here by this tool. It was mentioned on the amanda-users list a few months back that the amanda tree had been checked by coverity, and the 2 bugs coverity found were promptly fixed.

    Thats not to say that as new features are added, new bugs haven't been too, but to actually call amanda a truely buggy application does stretch this users belief a wee bit. I'm currently running a 20060424 dated snapshot of the 2.5.0 tree, with no hiccups at all.

    --
    Cheers, Gene
  • Firefox is, once again, the most unstable program in common use [slashdot.org].

    The 1.5.0.4 version of Firefox was quite stable, if the Flashblock extension was installed. The 1.5.0.6 version is unstable again. The CPU-hogging bug is back!

    This comment posted from a copy of Firefox that is constantly using 2.8% of the CPU, even when all pages have been loaded, and there is no active content. That's 2.8% on the way to 70% or more, making it necessary to close Firefox and reboot Windows XP.

    There are some bugs found b [mozilla.org]

  • Types of bugs (Score:3, Interesting)

    by Dan East ( 318230 ) on Saturday August 12, 2006 @02:23PM (#15895050) Journal
    After looking at some of the results from the Firefox sources, I see that "bugs" include unreferenced variables and dead code that never gets executed.

    It looks like most of the real bugs consist of not checking return values, the worst being routines that act upon an object allocated by another routine without checking for null pointer.

    Dan East
    • After looking at some of the results from the Firefox sources, I see that "bugs" include unreferenced variables and dead code that never gets executed.

      Checking for things like this is useful, though: If you've got code in software that doesn't get executed, or a variable that never gets used, you've almost certainly got something in there you didn't intend. Since bugs are when programs don't behave the way they're intended to, code that isn't quite what you expected is a good place to look...

      -F

  • Homeland Security Tested XMMS?! (Score:3, Insightful)

    by Pulsar ( 4287 ) <champ77NO@SPAMhotmail.com> on Saturday August 12, 2006 @02:31PM (#15895067)
    XMMS, a multimedia/mp3 player was tested as part of what the article calls a "$1.2 million, three-year grant [the Department of Homeland Security] awarded to a team consisting of Coverity, Stanford University and Symantec Corp" that was setup to "reinforce the quality of open-source programs supporting the U.S. infrastructure".

    40 programs were tested. 40 open source programs. Not even all the programs installed by, or regularly used on, a default install of a particular distro or two; just 40 programs. I thought maybe these 40 were just the first 40 tested, but the original announcement of the award of the grant states that 40 programs would be tested.

    And yet they didn't test BIND? ssh? Also, PostgreSQL is on the results list, but MySQL isn't? Did Homeland Security put this list together?! Using a dartboard and a list of open source applications, or what?!

    This seems like a great software package, and I'm glad that Homeland Security acknowledges that "much of the critical infrastructure runs on open source", but I could think of a few other ways they could've spent $1.2 million, or at least a few other applications they should've tested before they got to XMMS.
  • The software industry has been focused QA for pretty much its entire existance. This software they are using is just a testing framework, albeit an intricite one. Unit testing, code coverage, black box, system tests... all marketing buz words for excersizing your software. If you are'nt familiar with the software dev qa process, here is how it generally works in production:

    Hiring dozens of QA people to discover discover a broken error path that a developer could have fixed in 5 minutes is inefficant an

  • Coverity on Windows? (Score:4, Insightful)

    by Money for Nothin' ( 754763 ) on Saturday August 12, 2006 @06:41PM (#15895937)
    "Coverity was also run on the Windows source code. Unfortunately, the 32-bit integer iterator in Coverity was 1 count too small to store the count of the number of bugs found, and so Coverity's counter rolled-over, showing that Windows actually has -2,147,483,648 bugs. Microsoft employees were ecstatic at the results, and Steve Ballmer was said to be seen dancing in his office, yelling 'developers, developers, developers, developers!!'."

Slashdot Top Deals

THEGODDESSOFTHENETHASTWISTINGFINGERSANDHERVOICEISLIKEAJAVELININTHENIGHTDUDE

Close