Forgot your password?
typodupeerror

US Government Studies Open Source Quality 165

Posted by ScuttleMonkey
from the prefer-just-to-stay-off-of-dhs-radar dept.
anadgouda writes "US Department of Homeland Security has released a report on open source quality in an effort to study the security of open source. 31 popular open source packages were studied as part of this effort. From the article: 'Coverity's report, Stacking up the LAMP stack: a study of open source quality, was produced as part of a $1.24m, three-year DHS Science and Technology Directorate effort to evaluate and improve the security of open source.'"
This discussion has been archived. No new comments can be posted.

US Government Studies Open Source Quality

Comments Filter:
  • So, (Score:5, Interesting)

    by Eightyford (893696) on Saturday March 04, 2006 @02:35PM (#14850700) Homepage
    So, does anyone have the numbers as to how much of the government uses open source? Is it mostly an applications thing (OpenOffice) right now, or are Linux and the BSDs much in use?
    • I would say more in the vein of Linux/BSD at least so far as the NSA having their own security oriented distro which is availible for download on their site.
    • Yes (Score:5, Interesting)

      by jascat (602034) on Saturday March 04, 2006 @03:21PM (#14850847)
      While not used on every desktop, I know of a lot of F/OSS being used everyday in the military. It would be stupid to not use it. Why would companies like Redhat and Novell spend money on getting their software certified to run on classified systems if it wasn't going to get used? While we may be selling out to Microsoft a lot, there are times when those of us who know better manage to convince the decision makers of the right tool for the job. In some cases, it's a MS product, in others, it's something else.
    • It depends on what it's for. The vast majority of DoD machines have Windows, but there are rome redhat boxes around as well. I've only seen OS programs on the Linux boxes beyond seeing Firefox every once in a while...
    • I know that the personnel boxes that the military uses are Unix based
    • The government is somewhat scared of OpenSource, especially in the government secrets world. It doesn't make much sense to be scared of OpenSource, but the argument that has been given time and time again is that anybody can look at the source code and hack into the system. This pertains more for the smaller projects that would be useful in the development on some government made software product. There are a few Linux distributions on the "safe list" and also OSX is on that list too. I think the argume
    • Re:So, (Score:4, Interesting)

      by egypt_jimbob (889197) on Saturday March 04, 2006 @07:52PM (#14851759) Homepage Journal
      Speaking as a student about to graduate and go into Federal Civil Service as a penetration tester, I can tell you that all of the agencies with which I have interviewed use mostly Linux. Well, all of the agencies that are actually good at what I want to do.

      So far it's been mostly gentoo from what i've seen, but there are some places that have to use RedHat because their management said it has to have 'support.'

      Bear in mind, however, that the places i'm interviewing are hardcore hacker groups, so this may be (and probably is) completely off the norm.
      • So far it's been mostly gentoo from what i've seen, but there are some places that have to use RedHat because their management said it has to have 'support.'

        The need for official support is obvious, even if in reality it ends up being provided by the on site local admins. No need to write it down in quotes and roll our eyes. Official agencies have to have somebody accountable, it's part of justifying the spending of the public dollar.

        As for Gentoo, sorry, but it makes little sense why anybody would choose i
    • by AHumbleOpinion (546848) on Sunday March 05, 2006 @11:37AM (#14853763) Homepage
      The US Navy replaced Sun with Yellow Dog Linux, originally on Apple hardware and now on some other PowerPC based hardware, for sonar processing on subs.
  • by Jeremy.DeGroot (878927) on Saturday March 04, 2006 @02:43PM (#14850708)
    I think it's great that the government is backing this kind of study, and I think the the high marks a lot of packages received will really be a boon to the OSS movement. I think the part of TFA that excites me the most though, is this:
    Coverity evaluated 15m lines of open source code with Stamford University's Computer Science Department. The report has identified bugs that can corrupt a machine's memory space, memory leaks, buffer overruns and crashes. Coverity said it would now engage with open source developers to improve code, and identify potential reasons for why some projects have more bugs than others.
    If they're going to take their comments back to the communities that develop the software, then this could give the development communities a lot to work on and improve, and that could give us some greatly improved software in a year or two's time. I think work like this is the real strength of Open Source, and I hope to see more of it in the future.
    • by Anonymous Coward on Saturday March 04, 2006 @03:13PM (#14850812)
      The parent is wasting valuable time on Slashdot that should be spent finalizing his Independent Study project for the College of Wooster [wooster.edu]. He has precious little time left.
    • I wonder how many of the potential suggestions have been made by the OpenBSD crew, and already rejected....
      • by Anonymous Coward
        Actually two of the OpenBSD developers worked for Coverity last I heard (i.e. Ted Unangst and Peter Hessler). This probably gives them some influence ...
    • Or, you could say that this is the strength of taxes being used to fund public science, rather than the strength of open source software. Now imagine if the estimated end-cost of the Iraqi invasion (US$2 trillion over the years + 3,000 US citizens + 30,000 Iraq citizens) was being put towards public science?
  • by BigBuckHunter (722855) on Saturday March 04, 2006 @02:44PM (#14850710)
    This study would be extremely valuable if they had submitted BZilla bugs for each and every defect they encountered. It's hard to tell from the article whether they did or not. One thing that I have learned from running ~arch in Gentoo is that if you don't submit bugs, things aren't going to get fixed.

    BBH
    • by Too many errors, bai (815931) on Saturday March 04, 2006 @03:04PM (#14850780)
      If these packages are used within the government, the security holes discovered are probably kept secret. National security and all that.
    • I hope they looked at DJBDNS and QMAIL.

      All software should be that good.

      If they found bugs in Bind, I'm not iterested in the rest of the report. That's just pork.
    • RTFA (Score:4, Interesting)

      by Night Goat (18437) on Saturday March 04, 2006 @03:35PM (#14850882) Homepage Journal
      From the article, which I'm SURE you read:

      Coverity evaluated 15m lines of open source code with Stamford University's Computer Science Department. The report has identified bugs that can corrupt a machine's memory space, memory leaks, buffer overruns and crashes. Coverity said it would now engage with open source developers to improve code, and identify potential reasons for why some projects have more bugs than others.

      Yes, the folks who ran the tests plan to submit their findings to the developers to help squash bugs.
      • I'm not sure what "engage with open source developers" means... Not just because they used the word "with", which was unnecessary and hard for me to parse. It doesn't necessarily mean that they're itemizing and reporting the defects. It may be some foo-foo conference where they review coding practices and plug some form of SDLC CM/EM/UAT crap. I hope that is not the case, and that we actually get something constructive of ot this. Most of us have been though ISO/Six Sigma/Sas70 audits before and seen n
        • I'm not sure this is a good thing for FLOSS. In military usage, "engage" means "fight", as in "We engaged the enemy at 09:00 and killed them all."

      • Awesome! OSS needs more government funded projects to find bugs and security issues. All those "experts" who kept bitchin that noone would spend the time or money finding security holes in OSS should go shut the hell up and go back to writing brochures for Microsoft.
      • This definitely adds weight to the "more eyes make bugs shallow" principle of open source.

        How many closed-source applications would get this sort of helping hand?
    • Yes. (Score:2, Informative)

      by Anonymous Coward
      I'm involved in one of the F/OSS projects that Coverity analyzed; and yes, they were co-operative with the dev team in sharing their insights.
    • "This study would be extremely valuable if they had submitted BZilla bugs for each and every defect they encountered."

      The article seems to suggest that the authors want to help with processes, rather than individual bugs.

      That seems like a much better long-term idea, especially if (and this seems likely) they analysed a sample of code.

      If someone analyses 1000 lines of code from a 100000 line project, then they'll have a fairly good idea of what processes (e.g. audits, code reviews, patterns) can help the te
      • They would've had to look at at least 50,000 lines of code. Since they found 32 hundredths of a defect in 1,000 lines of code. (They could've found 16 defects in 50,000 lines). Wait, that doesn't seem right. Anyone want to enlighten me?
  • by toddbu (748790) on Saturday March 04, 2006 @02:44PM (#14850712)
    I feel very conflicted by this report. On the one hand, I'm happy to see a report that favors open source. On the other hand, in the wake of the Katrina political fallout, it's difficult to say whether this report helps or hurts. The last thing LAMP needs right now is to get caught up in Brown/Chertoff/GWB affair. The only thing worse would be to have the UAE issue a similar report. :-)
    • I don't see a reason to feel conflicted, unless you believe that some people/companies/institutions are pure evil 100% of the time or pure good 100% of the time. The world is a bit more nuanced than that.

      I'm sure if you looked at the lives of Stalin, Attilla the Hun, Saddam Hussein, and other despicable people you'd find that as bad as they were, they did *some* good. The opposite is true for Pope John Paul II, Ganhdi, and JFK.

      My own philosophy is to praise people/companies/institutions when they're good (n
      • I think it's a matter of perception rather than a strict good-vs.-evil accounting. If your work is praised by a source widely considered to be incompetent and/or corrupt, then people will perceive your work as worse, not better, regardless of its actual merits -- or, for that matter, how justified the praise itself may be.
      • But when you are judging an action that is proposed to happen at sometime in the future, you are always operating with incomplete information, and information that is biased in the favor of whoever released the information. In such cases the course of wisdom is to examine the proposal in the light of your best guess of what the motives are, based on past actions of the agencies involved.

        If someone has proven untrustworthy in the past, it's not wise to trust their promise about what they're going to do...bu
      • My own philosophy is to praise people/companies/institutions when they're good (no matter how bad they are normally) and condemn people/companies/institutions when they're bad (no matter how good they are normally).

        You must be new here.
    • That's a really ridiculous thing to say. The US government is supposed to be set up as a meritocracy. The idea is that there are career beaurocrats who sit in their jobs all of their life, independent of who in power. The branches of government like the GAO, NASA, the President's Office of Management and Budget are all known for this. Not everything that goes on in washington has to do with politics.

      And frankly, i find it pretty weird to think that an operating system or software development movement
      • by toddbu (748790) on Saturday March 04, 2006 @03:38PM (#14850892)
        The branches of government like the GAO, NASA, the President's Office of Management and Budget are all known for this. Not everything that goes on in washington has to do with politics.

        You can't really be that naive, can you? Take the OMB for example. There's a big debate [ombwatch.org] going on about whether OMB should use static scoring or dynamic scoring. It doesn't really matter which one you prefer, but I can tell you that in the current political climate it makes a *huge* difference. Democrats prefer static modeling because then they can argue against tax cuts. Republicans favor dynamic modeling to support a "trickle down" effect. But the idea that somehow OMB is neutral is ignoring reality. Even if they don't intend to favor one party or another, the fact is that there is no action that they can take that won't benefit one group or another.

        Interesting that you should mention NASA. Their very existence depends on the support of the aerospace community and the regions of the country that benefit from NASA centers. They are very good at using their influence to get what they want. Even if you could claim that they don't favor one political party over another, they are still very skilled at using political influence to their advantage.

        • That's not a boolean statement. There are shades of apolitical neutrality. Obviously, the OMB, as a direct branch of the administration is certainly going to feel more pressure than the GAO for instance. That still doesn't mean that all of the research and statistics that come out of the OMB are going to be slanted for political purposes.

          That aside, my point about casting linux with in a partisan political still stands. One might be able to cast open source software, in an anti-business light, but tha
    • The only thing worse would be to have the UAE issue a similar report. :-)

      Oh no too late :O [linux.de]
    • There is no relationship between this study and Katrina. The disaster people work in a different office, down the hall. Would you like me to transfer you? Hold on....
    • was there any point to your post other than attempting to incite a flame war?
      • Why would you think I was trying to incite a flame war? Because I noted that there is a current political firestorm over Homeland Security and the UAE? The whole point of my post is that it's easy for good data to get lost in political debate. I think your post proves my point.
  • by Mancat (831487) on Saturday March 04, 2006 @02:44PM (#14850713) Homepage
    Open-source software is a serious threat to this country. These terrorist schemes, or "development projects," as the terrorists refer to them, are designed to rot away the core values of our great nation that we hold so dearly. One in particular, known as "Linux," is especially suspect. It is "developed" by terrorists worldwide, many of which are communists, and many of which do not even support our great commander in chief! It is apalling! How can we trust the security of our nation to these rogue "developers?" Surely they may have hidden devices in their programs, hidden in elaborate matrices of computer programming, that when activated by the terrorists, will disable the software and send them all of our secret data! It can only be expected.

    The terrorists are cunning, they are secretive, and they will destroy us if they have their way. This world-wide "open source" terrorist movement must be deconstructed and eliminated. There is no other way to protect our Great Nation! We say to you, as the purveyors of truth and all that is good, avoid this "open source" and its proponents like the plague! They wish to destroy everything we hold dear. You, my good American, are the first line of defense. Report users of "open source" to the authorities. Gather any information on them that you can. You may even consider running their dastardly "software packages" in your own free time, so that you may come to know your enemy - for knowledge is the greatest tool that we have in this fight.

    Stand proud, my fellow Americans, and beware this new emerging beast. It will surely be the end of us all if we do not take action now.

    Quoted from President George W. Bush's State of the Nation Address, January 2007.
  • by boa13 (548222) on Saturday March 04, 2006 @02:49PM (#14850725) Homepage Journal
    One would expect that being about open-source and all, and with a purpose of helping open-source developers improve the quality of their code, they would publish the report on a governement website somewhere. C'mon, where's the link?
  • by hihihihi (940800) on Saturday March 04, 2006 @02:52PM (#14850733)
    the report have a better coverage on this page: http://www.eweek.com/article2/0,1895,1909946,00.as p [eweek.com]

    from this TFA:
    "Anti-virus vendor Symantec Corp. is providing guidance as to where security gaps might be in certain open-source projects."

    PS:i am not sure if it has been published on /. or not
    • Well, it *sounds* good ... but Homeland Security? Symantec? I think I'll reserve judgement for awhile. And Stanford has also got a mixed reputation WRT openness. Before I even trusted their intentions I'd want to go over the contract with a lawyer. Sometimes they're good guys, and other times...well, lets just say that I'd like to reserve judgement.
  • by sreekotay (955693) on Saturday March 04, 2006 @02:52PM (#14850735) Homepage
    I've always thought it VERY odd to think about "Open Source" as a thing.

    It'd be like saying: We studied the quality of software compiled with the Watcom 10.0 C++ compiler. "Open source" cuts across so many levels of skill and projects. You can pretty find projects that support (or destroy) whatever thesis you'd like to put forward

    Even more, somebody pays for the development of the software, one way or another.

    This artlice (from ONLamp) http://www.onlamp.com/pub/a/onlamp/2005/07/21/soft ware_pricing.html [onlamp.com] really puts into better perspective. Basically, it says ALL software can be deconstructed to being about the service (at least so long as the technology curve continues, in practice, to limit its lifespan).

    --
    graphicallyspeaking [kotay.com]
    • It's a lot more difficult to study the bugs in closed source code and get a bugs per thousand lines of code metric out of it. That is probably why they're doing the testing on OSS.
    • We studied the quality of software compiled with the Watcom 10.0 C++ compiler.

      That is perfectly logical. Software that comes OUT of a compiler should certainly be tested for quality. Watcom processes source code, and produces a resulting change, so it's valid to ask questions about that. Likewise, Open Source is a process, with its own unique qualities and product attributes. Also, it's an ALTERNATIVE process to the main ones used to develop software, so the idea of evaluating the different outcomes fro

    • I disagree. Open Source is not a thing, it is a process. A process that's of interest because of its products.

      That's not to say it's easy to study in a way that you can use to make decisions about open source product A and closed source product B, but it's far from impossible.
  • "...has effectively given the Linux, Apache, MySQL and Perl/PHP/Python (LAMP) stack a healthy rating. LAMP "showed significantly better software quality" above the report's baseline with an average of .32 defects per 1,000 lines of code, according to Coverity. The average for open source projects analyzed is .42 per 1,000 lines." What would be interesting to know is how they determined a baseline of .32 defects per 1000 lines of code as their baseline, and how so called commerical products, like Oracle
  • Compare with... (Score:2, Interesting)

    by Anonymous Coward
    ...New Zealand's recent analysis of open source [e.govt.nz], which focuses on legal issues.
    • Re:Compare with... (Score:3, Insightful)

      by cyber-vandal (148830)
      25 There is a risk that open source software contains functional defects, or breaches a third party's intellectual property rights (e.g. where it contains code misappropriated from proprietary software or functionality in breach of a patent). The absence of warranties and indemnities in most open source licences means the licensee bears this risk. This can be contrasted with the protection usually available under commercial software licences.

      That made me laugh.
  • is good for the gander?

    I wonder what "bugs that can corrupt a machine's memory space, memory leaks, buffer overruns and crashes" have been uncovered by looking at the source code of closed source softw... oh. wait. no source. heh.

    This might well mean that open source software will, at some point in the future, be considered more secure and well-written than comparable commercial closed source software even by government or PHBs.

    You have to wonder about the difference in "errors per thousand lines of codes"
  • by Old Duck (957936) on Saturday March 04, 2006 @03:39PM (#14850898) Homepage

    An interesting study was done by the U.S. Military (the Airforce, I believe) concerning Open Source and it's place in the department of defense, though it is written in such a way to be useful to non-military personnel and applications. It is a similar, yet IMHO, a more interesting read than the parent.

    The report can be found as a PDF at [af.mil]http://www.stsc.hill.af.mil/crosstalk/2005/01/0501 Tuma.pdf [af.mil]

    • by Anonymous Coward
      Upon reading the PDF it struck me that if an organisation like the military wanted to use OSS in a more secure fashion, then the use of closed locked down binaries of the code like a default Linux secure network setup is the best option. The problems arise when the individual nodes can be modified willy nilly by malicious code. If you do not include a compiler on the nodes and make sure that binaries cannot be installed by users then you have a blueprint for bullit proof security. Given that the code that i
  • What is normal? (Score:2, Insightful)

    by CAPSLOCK2000 (27149)
    FTA:

    LAMP "showed significantly better software quality" above the report's baseline with an average of .32 defects per 1,000 lines of code, according to Coverity.
    The average for open source projects analyzed is .42 per 1,000 lines.

    Does anyone have any factual data on what is "normal" (accepting all the problems of counting lines and bugs in the first place). I've seen estimates range from 2 to a 100 per 1000 lines.

  • Next time give that money back to us and write "USE OPENBSD" on your report. Better yet, just give them the money, and they'll actually do security stuff with it.
  • Wow (Score:2, Funny)

    by ROOK*CA (703602)
    Three years, $1.24 Million, and what do we got .....

    The envelope please ...

    "LAMP "showed significantly better software quality" above the report's baseline with an average of .32 defects per 1,000 lines of code, according to Coverity. The average for open source projects analyzed is .42 per 1,000 lines."

    Wow, LAMP is a pretty damn high quality stack after all....gee thanks Captain Obvious, we didn't really need those tax dollars for anything anyways. :)
  • superb! (Score:5, Funny)

    by macsox (236590) on Saturday March 04, 2006 @04:45PM (#14851112) Journal
    if there is one group of people i trust to be able to accurately identify a quality product, it's the government.
    • As opposed to who? Microsoft? Here in the UK Margaret Thatcher preached a similar mantra that government organisations are useless, inefficient and bureaucratic. So she privatised like a demon and now we have public services that are not only useless, inefficient and bureaucratic but now also largely unaccountable even though people still depend on them as much as ever. The profit motive doesn't automatically make an organisation better.
  • Hmmmmm, wonder what Vista would look like under that scrutiny?...
    Hmmmmmm.... Hey, I have a thought: if Microsoft does as it says and allows the Gov't to view it's code (without releasing it), should not this standard of examination be applied to Microsoft's software too so that we could have a better idea of just what level of quality we can expect from the private sector?
    • If Vista has 40,000,000 lines of code and 10,000 bugs were found (that's 50 fixes in each Windows Update, every week, for four years), they'd still be better than 0.32 defects per 1000 LOC.

      I've no idea how many lines there are in Vista (or, for that matter, how you count them), but the rumours say that Windows XP is about 40M LOC.
  • Same Old Math Error (Score:2, Interesting)

    by oldCoder (172195)
    These guys just can't think straight:

    LAMP "showed significantly better software quality" above the report's baseline with an average of .32 defects per 1,000 lines of code, according to Coverity. The average for open source projects analyzed is .42 per 1,000 lines.

    So if LAMP open-source is simply more verbose than other kinds of open source, the number of bugs per line of code can go down? How about just adding a million lines of bug-free but totally bogus code to your project -- and completely winnin

    • "How about just adding a million lines of bug-free but totally bogus code to your project"

      If it were that easy to write a million lines of bug-free code, we'd all be doing it. Bogus code is *MORE* bug prone than application code. Why? Because it's never tested.

      Sure, in theory, people could just add a bunch of lines with just semi-colons. However, in practice, the testing agency would notice this and come up with a screen. Anything more complicated than empty statements is prone to error.
    • by tricorn (199664)

      The real problem isn't bogus stats caused by line inflation. The real problem is that it only finds certain types of bugs. If a bug causes an incorrect result or improper behavior, but doesn't cause a memory leak or the like that crashes the program or system, then it isn't being found. It also isn't finding STUPID code - code that works but is ridiculously convoluted, slow, difficult to modify, redundant (writing 5000 lines of code to do some string manipulation and parsing that could be done just as ea

  • Well, at least it can be seen that there is overwhelming bias at slashdot. Not that I care, since I still read the news here.

    If any MS (or should I say M$) product were to have been put in an article like that, the mobs would have screamed for Gates's head. However, since it is the all-powerful-silver-bullet-snake-oil open source, all I see are excuse makers and doubters. If anyone is to even take themselves seriously, they must be at least OPEN to the idea that something they believe in is not perfect,
    • Well, at least it can be seen that there is overwhelming bias at slashdot.

      - Saying that one race group is inferior to another constitutes a "bias": correct.
      - Saying that some software is better than other software constitutes a "bias": incorrect

      The two are not analogous. The flaw in your argument is the implicit assertion put forth that "all software is created equal" (so to speak) and that any preference of some software over another must therefore constitute a bias. Here's a cluestick for you: Softwar

  • SE-Linux (Score:2, Interesting)

    by Anonymous Coward
    Why no mention of SE-Linux?
    One agency study.
    1.5 million dollars spent.

    How much did the NSA spend developing SE-Linux?
    Must have cost more than 1.5 million. And that is now at the core of Linux.

    Yes many in the US Government are aware that Open Source software rocks.

    Impeach the Liar
  • by Morganth (137341) on Sunday March 05, 2006 @12:27AM (#14852518) Journal
    I know that there is a Stamford University, and everyone always jokes that it's for people who want to pretend they went to Stanford, but, this just makes things really confusing. The Register article says Coverity used a verifier from Stamford University, when really the program came from Stanford [coverity.com]. In fact, AFAIK, UCONN-Stamford doesn't even have a CS department.
  • Where's the Beef? (Score:3, Insightful)

    by PhYrE2k2 (806396) on Sunday March 05, 2006 @01:17AM (#14852626)
    To quote the Wendy's commercial, "Where's the Beef?".

    No seriously! Where's this article? I'd imagine three years and 1.25 million dollars would produce a hefty article. I'd love to give it a read! "US Department of Homeland Security has released a report on open source quality"- so where's the release?

    It cites one or two figures, and throws around lots of buzz-words, but there's no comparison? No information? No study of reliability? Nothing at all.

    PS: As a side-note, if they 'studied' 15 million lines of code over three years, and were able to identify defects, shouldn't we be seeing a nice patchset coming from Coverity sometime soon... Think about it. It's easy to tell someone else to fix it, but a good part of OSS is giving back.
  • It would be very useful if they could do some of the following, if in fact DHS was supposed to be in this business which I doubt, it is really a very gray area. But they seem to have free time on their hands so a wish list:
    • Tell authors about bugs they find, as they find it
    • Submit bugs via the project's bug submission system
    • Develop a bug submission standard object format and open testing methodology, maybe even a server and some ontology to help automate this stuff?
    • Teach developers ways not to make thos
    • Thanks for your very interesting comment, it sounds like you live an exciting life! Point well taken. Perhaps government(s) will start to outreach more to open source software developers and this is just the beginning of a good thing, and granted perhaps one of the best and least destructive things DHS could choose to do.

      It seems to me that both the DHS and the open source community would benefit from a broad discussion of how DHS can and should contribute, in particular if they are spending millions mayb

  • This article is kind of dump. It compares LAMP to everything else FOSS.

    I don't need that, I need to know how FOSS compares to Proprietary Software

In order to get a loan you must first prove you don't need it.

Working...