Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×

Debugging Expert Wins ACM Dissertation Award 83

Posted by ScuttleMonkey from the just-a-programmer-with-a-really-big-shoe dept.
An anonymous reader writes "The Association for Computing Machinery (ACM) is reporting that Ben Liblit has been awarded the 2005 Doctoral Dissertation Award for his study on understanding and fixing software 'bugs' in the real world. From the article: 'Liblit's dissertation proposes a method for leveraging the key strength of user communities - their overwhelming numbers. His approach uses sparse random sampling rather than complete data collection for gathering information from the experiences of large numbers of software end users. It also simultaneously ensures that the observed data is an unbiased, representative subset of the complete program behavior across all runs.' Slashdot broke the story on this research back in 2003. Apparently the project is still going strong."
This discussion has been archived. No new comments can be posted.

Debugging Expert Wins ACM Dissertation Award More

Debugging Expert Wins ACM Dissertation Award

Comments Filter:
  • they're not bugs, they're features.

  • blargh (Score:1, Offtopic)

    by MarkPNeyer ( 729607 )

    'leveraging'

    *tears out own hair and screams*

    • Comment removed (Score:4, Funny)

      by account_deleted ( 4530225 ) on Tuesday March 21, 2006 @11:56PM (#14969489)
      Comment removed based on user account deletion

    • Re:blargh (Score:3, Informative)

      by Anonymous Coward
      tr.v. leveraged, leveraging, leverages 1.
      a. To provide (a company) with leverage.
      b. To supplement (money, for example) with leverage.
      2. To improve or enhance: "It makes more sense to be able to leverage what we [public radio stations] do in a more effective way to our listeners" Delano Lewis.

      So listen and listen good all you academic paper writers: unless what you really mean by "leverage" is "improve", don't use it.

      "Liblit's dissertation proposes a method for leveraging the key strength of use

      • Re:blargh (Score:4, Interesting)

        by Tony-A ( 29931 ) on Wednesday March 22, 2006 @01:25AM (#14969751)
        The root word is lever and the basic idea is that you use something under your control to effect control over what would normally be outside your control. Like a very long handle on a pipe wrench.

        The money aspect you refer to has to do with debt financing whereby you manage to use your equity to finance something larger than your equity. I don't think the article is referring to corporate finance.

        In a perfect world you would use a few people who would recognize and fix the bugs. These people would never talk to the users. They would have no need to and neither would gain from the experience.
        In the world that I exist in, users are the ones who spot the bugs, specifically the circumstances under which the bugs exhibit themselves. I use my user's eyes to leverage {user's eyes, my skills}.

        If all you mean is "improve", you would not use a word which essentially demands a discrepancy in the metrics between cause and effect.

        b. To supplement (money, for example) with leverage.
        If you add money to an account because of a margin call, does this increase or decrease your leverage? That is a horrible excuse for a definition.
         

  • Sounds like Doc Watson (Score:4, Insightful)

    by BadAnalogyGuy ( 945258 ) <BadAnalogyGuy@gmail.com> on Tuesday March 21, 2006 @11:41PM (#14969415)
    No, not the wild-eyed madcap scientist from Back to the Future. Doctor Watson is an OS service present in Windows that monitors the running process list for terminal assertions. When a program hits an exception that it can't handle, it terminates immediately and Doctor Watson is on the scene to read the last gasps of the process before its bits get blasted. Microsoft even came up with a way to harness this to allow users to send real-time feedback to Microsoft HQ whenever a crash occurred in a program. No one I know ever sends that data back, but I'm sure someone must have once.

    The current idea seems to be tracking the same termination events in the same way as Doctor Watson and sending the relevant data back to UWisc without informing the user. It sounds like a good idea, but I doubt it is in Liblit's power to fix Windows OS bugs.
    • Microsoft even came up with a way to harness this to allow users to send real-time feedback to Microsoft HQ whenever a crash occurred in a program. No one I know ever sends that data back, but I'm sure someone must have once.

      My Grandfather was having trouble running FlightGear under windows and sent one of those to Microsoft. Even got a response telling him it was a driver problem.

      I send Firefox errors, just to give them some hope :)

    • Re:Sounds like Doc Watson (Score:5, Informative)

      by Benoni ( 132028 ) on Tuesday March 21, 2006 @11:57PM (#14969493) Homepage
      sending the relevant data back to UWisc without informing the user.

      Informed participation is a really big deal for me. No user should ever find themselves participating in the Cooperative Bug Isolation Project without their knowledge. Opt-in is explicit and revokable, and if the opt-in system runs into trouble of any kind, the fallback position is no data reporting at all.

      The whole thing collapses if users don't trust me. So I've taken every measure I can think of to ensure that they can. Please see the relevant project page for more details about privacy matters [wisc.edu].

      It sounds like a good idea, but I doubt it is in Liblit's power to fix Windows OS bugs.

      Working on it! Check back in with me in a few years ... maybe less. :-)

      • The installation of CBI is implicit consent to such monitoring, of course, and I didn't mean to imply that there was no consent involved at all.

        However, asking us to read 170-odd pages of your dissertation is a little much. Would it be possible to describe the data collection system, how reports are generated and if the reports are sent automatically or as in the case of Dr. Watson sent with user approval. Also, what types of bugs you found using your statistical methods, as well as what types of bugs you

        • Re:Request for more information (Score:5, Informative)

          by Benoni ( 132028 ) on Wednesday March 22, 2006 @12:17AM (#14969559) Homepage
          However, asking us to read 170-odd pages of your dissertation is a little much.

          Hey, it's a real page-turner. Well, it has pages and they turn, at least.

          The other questions you ask are all good ones, but a bit much to address in a Slashdot comment. Please see the project home page [wisc.edu] for more information. The "Learn More" [wisc.edu] page may answer some of your questions, and there are additional drill-down pages from there with even more technical material on selected topics.

          Please understand that I don't mean to brush off your insightful questions. They are just questions for which satisfactory answers are hard to give in a sentence or two.

        • asking us to read 170-odd pages of your dissertation is a little much.

          Why? If you're really interested in that area, that's not much material and it's important research. I know that I'm going to read it just because it's interesting.

          But if you're not deeply interested, you will be able to pick the most interesting bits by looking at the table of contents, won't you? Or is that too much effort, too? Besides, the project has a Web site that is even referenced in the /. blurb -- did you even bother to lo

          • Well, we've got Ben Liblit RIGHT HERE! Right here in the thread! Replying to my question!

            Doesn't it make sense to ask him while he's here to discuss the topic in a simplified manner since he's the world's leading expert on the topic? This is a discussion forum, so being able to hear him express the concepts allows us to participate with him in a two way transfer of ideas.

            The alternative is to write everything down and simply refer to documents instead of engaging people in conversation.
            • I don't think that a request to reiterate the main points of his research is a good start for a sensible discussion here. (Not that I expect many sensible discussions on /.) Especially not if some of these questions are answered on the homepage and the `About this project' page. Your telling that the data is sent without user consent -- when the first paragraph on the home page tells that data is sent back -- and then slowly backpedalling is not a good start for a /. discussion either.

              If you would have co

              • I see. I must inform myself by reading his dissertation so that when I come back in several hours with questions he will be able to field them and we can all have a rousing conversation.

                I'll keep that in mind the next time I have an interest in something and have the opportunity to have direct access to an expert and have essentially carte blanche to ask anything. Read up on it first, then hope that the expert hangs around while I'm busy informing myself.

                As for the thrust of my main post, it was not about

    • Re:Sounds like Doc Watson (Score:5, Interesting)

      by Anonymous Coward on Wednesday March 22, 2006 @12:14AM (#14969551)

      No one I know ever sends that data back, but I'm sure someone must have once.

      Plenty of users do. There's a great blog posting by Raymond Chen called There's an awful lot of overclocking out there [msdn.com] where he talks about investigating some of these "Watson" crashes.

      The crashes were impossible - instructions like

      xor eax, eax

      Turns out unscrupulous vendors were selling overclocked computers without informing buyers. Pretty cool article.

      • The crashes were impossible - instructions like
                xor eax, eax

        Watch out -- you may have just reverse-engineered Sony's latest DRM enforcement mechanism!

      • The crashes were impossible - instructions like

        xor eax, eax

        How is that impossible? It's a single-byte, single-clock, u/v pipe op to zero eax... How do you (or the original author of the comment) do it? mov eax,0? *snicker* That'd certainly explain some of the Windows bloat...

    • If you even just skim the page, you'll see that only the fact that it sends a feedback report is similar. Most of the project consists of 'sparse' weighted sampling of instrumented code, research into what are useful metrics to use to instrument the code, and how to correlate the collected data with patterns to auomate finding bugs.
    • My main issue with the good Doctor is that most of the time, when I had a program that crashed and invoked him, it wasn't a Microsoft product. Typically, it's because I was working on a programming assignment and it 'burped.' So unless Microsoft was willing to help me debug my homework, I didn't see much point in sending the data on to Redmond.

      Not that I mind sending back data when it can be useful; if someone is going to look at the error logs, memory, etc., and try and make it so that it won't crash aga
    • It goes back much further than that. See "The ALCOR Illinois 7090/7094 post mortem dump" [acm.org], a famous paper from 1967.

      Automated dump analysis is an old idea in the mainframe world, but almost unknown outside it. The microprocessor world grew up with interactive debuggers and an early user-as-programmer assumption. This hasn't translated well to the modern software world.

      In the mainframe world, there have even been mainframes that recorded the last 64 or so branches using dedicated hardware, so that afte

    • The mad scientist from "Back to the Future" was named Emmett Brown, not Doctor Watson.
    • I always send my error records to microsoft, then either they fix it or they pass it on to the purveyor of the software. I've had a number of issues identified and remediated because of this feedback loop. There's no reason not to send the crash report to MS. It contains no data about your machine or it's contents. If MS passes the bug along, they don't identify the source. So where's the problem?
    • The mad scientist from Back the Future was Dr. Emmett Brown.

  • Check out FindBugs for finding bugs in Java (Score:5, Informative)

    by licamell ( 778753 ) on Tuesday March 21, 2006 @11:43PM (#14969423)
    This reminded me of work going at at UMD (University of Maryland, College Park). I know it's not quite the same thing, but I feel as though this is a good place to mention it and the slashdot community would appreciate this software. FindBugs is a very cool tool for finding bugs in java code. And no, I am not affiliated with this project, I just saw a talk on it a couple months ago.

    http://findbugs.sourceforge.net/ [sourceforge.net]

  • "Sir, is this a stand-up fight, or another bug hunt?"

    Seriously, congradulations, Ben!

  • Thank you, open source community (Score:5, Informative)

    by Benoni ( 132028 ) on Tuesday March 21, 2006 @11:49PM (#14969458) Homepage

    This research has been a wonderful collaborative effort, and many people deserve to share the credit. To quote from part of the Acknowledgements section of my dissertation [wisc.edu]:

    I am indebted to the many members of the open source community who have supported our work. My thanks go out to the many anonymous users of our public deployment, and to the developers of the open source projects used in our public deployment and case studies.

    So thanks, Slashdot, for helping me find those users (or helping them find me). The exposure was invaluable. And thanks, open source community, for your participation. I've benefitted greatly from standing on your massed shoulders. This could not have happened without you.

  • Ben Liblit is now an assistant professor at the University of Wisconsin-Madison. He joins a fantastic Computer Science department. Good luck Ben!

  • Closed versus open source comparison (Score:1, Interesting)

    by Anonymous Coward
    It would have been interesting to know the difference in number and type of bugs between a closed source product and an equivalent open source product... let's say the MS Office suite versus OpenOffice.org suite.

  • Heh... (Score:5, Funny)

    by the_skywise ( 189793 ) on Tuesday March 21, 2006 @11:55PM (#14969484)
    So somebody went and formalized the theory of "the users are the beta testers"...

    • Re:Heh... (Score:5, Informative)

      by Benoni ( 132028 ) on Wednesday March 22, 2006 @12:00AM (#14969501) Homepage

      Yes, exactly. The users are beta testers; we may as well admit it. I want to make them better beta testers. :-)

      • It has been my experience in software development that sharp users can give valuable feedback in the areas of usability and expected behaviors, but this is no substitute for trapping errors and logging failures on the user's machine when it comes to tracking down less visible faults.
      • having wrangled beta testers, I know that I totally spoiled the engineers at my last/psuedo-current gig by writing coherent and complete bug reports... sigh.

        (hi, Ben! ;)
      • Ahh...the many years I've spent trying to make managment understand.... And why Open Source makes so much sense, at least from a got broke/get fixed perspective. Still haven't figured out the financial incentives. :)

        • Still haven't figured out the financial incentives. :)

          If you mean incentives from the users' perspective, I like to pitch it this way. My statistical methods naturally tend to "learn" the most, most quickly, about the failures that happen most often. So the more a user participates, the more the developers' attention will be swayed to the bugs that user cares about. Thus, users can help steer bug triage.

          Open source bug trackers can work the same way. When you report bugs in some project's Bugzilla s

  • I know that in one particular http://www.kingdomofloathing.com/ [kingdomofloathing.com]game, they tend to follow this approach. Once a new feature is created, and debugged enough so that it's stable and doesn't break anything, the feature is released to the general populace. After all, once all of the important bugs are found, a thousand users will find the minor bugs through general usage faster than a small dedicated team of testers. Also, the time the testers save by not having to verify every single minor detail can be used
  • You mean the ACM is more than just a drinking club? The way they advertised themselves at my university, one wouldn't have thought so...
    • ACM itself is a real professional organization; however most of the student chapters are primarily drinking clubs. I had to disassociate myself after the new officers started running a porn business out of the office.
  • I think that automated unit testing is the future of killing bugs. In layman's terms, this involves a program trawling your code and automatically trying to break it. If done well, the system can replace some of your QA team, and QA goes a lot faster. I hadn't even heard of such a thing until I did some contract work for Agitar, one of the companies doing this stuff. Here's a link with an overview & screenshots:

    http://www.agitar.com/products/20051101-agitator.h tml [agitar.com]

    They only work with Java. Here is
    • Unit testing tends to only confirm that the program under test works as designed. It does not catch design errors or requirements errors. To catch these, you need to design a test that confirms that the system works as its supposed to work. Where "as its supposed to work" is some arbitrary, external criteria. You should also note that I said "system" and not "program" where system is some larger assembly of components including the hardware and user environment.

      You do better if you actually force the de

  • reminds me of my rules on reporting list outages (Score:3, Funny)

    by SuperBanana ( 662181 ) on Wednesday March 22, 2006 @04:40AM (#14970211)
    This reminds me of a method on reporting mailing list outages I devised back in 2001 or so.

    I told people we were switching to new software (Mailman)- and that if they got an error message or similar, to flip a quarter X times (I forget how many) and ONLY email me if they got all heads. I didn't want to get a couple dozen reports of the same problem, and I figured that if there were any problems, they'd affect a large set of the 1000+ users of the list.

    It worked brilliantly.

  • Guardian doesn't think so (Score:1, Interesting)

    by Anonymous Coward

    'Liblit's dissertation proposes a method for leveraging the key strength of user communities - their overwhelming numbers.`

    'Of all the myths that have grown up around open source software, perhaps the most pervasive is Eric Raymond's aphorism that "Many eyes make bugs shallow",`

    - Andrew Brown [guardian.co.uk] Dec 08 2005

  • I would like to see the data that they're collecting but I can't find it anywhere on their site. Am I just missing it?

    I learned my lesson with the cddb disaster: don't submit your own data unless you can mirror it yourself. Otherwise, if someone gets bored or greedy, everybody's hard work gets lost forever. Or, worse, it gets subverted to make profit for Gracenote.

    Mirroring is easy enough even if you have almost no bandwidth: use bittorrent. So, where are the submitted results?

  • Example: yesterday I had to solve a problem in the application I am developing for Bell that was absolutely independent of the programming platform.

    The QA reports that there is an error on the screen while they are trying to save some data. The error is intermittent, it happens for some data sets but not for others. Investigation shows the following:

    1. The data records that need to be saved must be first compared to existing data records for the same primary keys, if there are records, then the data is up
  • RANT #1
    I always think it's weird, something about the art and craft of getting better code out gets little notice, but some FireFox alpha (which is feature INcomplete, really only for extension developers) gets 200,000 messages. Mabe a third of them will be flames "(IE/FireFox/Whatever) is so buggy, you suck unless you switch to (whatever)". But tools to debug these get ignored. How much work is going into KDE vs GNOME, and even a group that wants to fork KDE (deity() help us).

    RANT #2
    I really think we're

Slashdot Top Deals

Love may laugh at locksmiths, but he has a profound respect for money bags. -- Sidney Paternoster, "The Folly of the Wise"

Close