Improving Software Configuration Management?

Elvis77 writes "I am managing a project looking at some software to help us with software change management. There are numerous good applications around for doing this and our purchasing is complete, but I have been amazed during my investigations at how many organizations rely on good manners, good intentions and good luck to manage the configuration management of their organization's software assets, even in light of the Sarbanes-Oxley Act of 2002 (SOX) that affects US companies (I am in Australia). Organizations outside of the USA, without SOX implications, are usually still concerned about the quality of their software. What do my fellow Slashdot readers consider to be the best practices for configuration management?"

Two paths

[gclef]

There are really two paths you're talking about here, and folks tend to confuse the two:

1) Software *development* change mangement. Meaning: tracking things like changes to software code.

2) Software *configuration* change management. Meaning: tracking changes to the configuration of the software. I presume you're talking about this one, but it's not completely clear.

Development change management is well served by tools of varying complexity from ClearCase to Subversion/CVS. Subversion/CVS seem to be very common, as they're free, but I've worked in offices using ClearCase before (not that anyone was terribly happy about it, though).

Configuration change management is much harder, as you're talking about managing the configuration of applications across potentially hundreds of machines. The tools for this vary widely, depending on how hard-core you want to be. They vary from CFengine to full-on provisioning systems (openQMS, for example)...none of them tend to be easy to set up or manage, which makes them less common.

Re:Use RCS for config files

[Asgard]

Now, go make that same change on 100 servers, without disturbing the other contents of the file.

RCS is great for single-server applications or one-off config files, but it doesn't scale really well to large applications.

More than code versioning

[$ASANY]

There's a whole other layer to CM, which is the issue tracking, requirements tracking, resource management and documentation management piece that probably is more critical than whether you can checkin/checkout changes and be able to back out a change.

We supported a government client where issues were managed with word document templates and emails. It was a disaster, with things getting lost and weekly meetings being the only times decisions were made. They were spending a boatload of money developing something that looked suspiciously like a hobbled version of bugzilla, so we recommended and got approval to stand up bugzilla for issue tracking. It's been a big success and has been expanded to handle requirements management as well.

The key was to set up "products" that matched operational areas for a product rather than thinking that a "product" was defined as a single deliverable. We set up the standard "product" as deliverable, with subcomponents which somewhat matched the functional areas where developer responsibilities were broken down. Then we established a "product" which was essentially an area where management issues could be handled, and had subcomponents for tester access control, requirements definition, external coordination issues and the like. So when we went into testing and found an issue that we decided to defer to a subsequent release, it was moved to the admin area and the requirements subcomponent. This kept policy and requirements control out of the hair of the developers and allowed parallel workflows for requirements, design, development and testing.

It's not a perfect tool for all of that, but it's pretty close to "good enough", and the price is right.

What would I add to this mix if I were God? I would wiki the work-in-process user's manual so developers could flag issues that should be addressed in documentation rather than in code. I would probably wiki and/or subversion the test plans, as word documents utterly suck for test plan management. And I'd spend some time with wiki "special pages" and bugzilla customized components to integrate it all -- linking and content sharing between a wiki and bugzilla would bring the solution set into the 90%+ range of requirements matching.

Coders rarely are the problem in software CM. It's the management, architects, functionals and tester coordination that really has the potential to kill a project. But if you can coordinate all of them well, the flow of functional requirement - use case - design - develop - test - debug - requirement generation & traceability comes together cleanly and raises a development process into a portfolio management/enterprise architecture execution process.

Not tools, but process

[Grab]

The general process required is that a change can be tracked from a customer request, through the design, to a code file, and back up the V (or round the spiral) to the testing/verification/validation used to make sure that the design and code do what the customer asked for. And then that you can track all the changes that went into a particular version build, which means recording what files you used to build it.

How do you do that? I don't care, and nor does Sarbanes-Oxley. We used to use Access databases at our place (and still do on some projects). Other projects use a tool called Visual Intercept, which I'd recommend people avoid like a rabid maneating tiger. Some projects have even used Excel (heavens preserve us!). Or simply use multiple text files (checked into your version control system of choice), one per customer request. Or Bugzilla or any number of other tools will serve your purpose. Tracking versions of files could involve some hugely complex build system, or it could just involve slapping a label across the files and doing "get all".

But if you don't grok the basic premise that every line in your code must be attributable to a customer request, then no tool you can buy will help you. This is a process you need to follow, not a magic-bullet tool you can install and everything will be fine. Repeat after me - There Is No Quick Fix For Quality. :-)

The most important thing you can do make sure your system works is therefore to review each change before it gets signed off to make sure changes are complete. That doesn't just mean reviewing that files are changed, but that they're all checked in under version control, AND that the change details (however you do it) are fully filled in. Even if there's automated fields to list all files and their versions in, review them anyway - there's bound to be cases where someone says "well I've got one test file listed, so that'll keep the tool happy, never mind about the other 20".

Grab.

drop the wooly language

[The Pim]

Any effort such as yours should begin with clear thinking, which is aided by purging yourself of such vague, overloaded, and literally nonsensical terms as "configuration management". In case you think this is a troll, note that the posters so far have given your message a range of completely unrelated senses. Describe precisely what you're trying to accomplish if you want useful answers.

Re:Not tools, but process

[PacketKing]

I will wholeheartedly agree with the parent post on many points.

Coming from someone who was a ConfigMgmt person for my company, I faced a lot of these issues. First and formost, get a plan, even a simple one, and get it written down. Then modify it as needed. Also, Label every time you build. Any decent source control tool will allow you to do this. Be consistent with your labels, and be clear with them. This way, when a need to build version x.y.z arises, you can get back to it. Trackability is key. Make sure your plan is built around it, because ultimately what you're looking at is being able to build at any time the same x.y.z that you released.

I also must second the phrase "There Is No Quick Fix For Quality". Do it right the first time (even if it takes a lot of time) because you won't need to go back and fix it later. This goes for your end product, but also for your process, as ripping up processes to replace them is tough. THis is not to say that you can't use prototype processes, but when you decide on one, stick with it.

Other tips & tricks:

- the wiki idea posted above is wonderful. I've worked with them for quite a few years now, and in a dev environment, they can be awesome. One caveat: make sure you get a wiki that does revisioning of the pages, one example would be TWiki (http://twiki.org./ [twiki.org.] This can be a godsend, just like figuring out code changes with CVS.

- I've managed 6 different revision control systems in my career, even being certified as a ClearCase admin. I'd have to say, Subversion is a pretty competent version control tool, and not terribly hard to learn. It's worth the time for the features it offers. As for the others? I'd say stick with subversion or CVS because:

1. the userbase is larger, therefore easier to find help
      
2. they're more than adequate for most development houses
      
3. they don't cost anything (compared to $3k per seat for ClearCase)
      
4. usually, you can get your data out of them in case disaster strikes (and don't think it won't, cuz it's happened to me more times than i care to think of).
   

You may have already failed the first test

[rossifer]

It doesn't sound as though you trust the professionalism of your staff. Not that you shouldn't provide the tools they need to be effective, but I notice that you're equating SOX and top-level control with software quality, which leads me to the conclusion that you don't trust your staff. In my experience, top-level control has almost nothing to do with software quality, while trust has been highly correlated with high quality products.

Step 1: Get the best staff you can. "10x" developers do exist, but you should be aiming for a staff of "3x" to "5x" developers of varying experience who work well together. The best developers won't really want to manage other people, but will want to be trusted with significant responsibility (i.e. they don't want to be fed detailed "specs"). Being good team members is at least as important as having top-notch skills.

Step 2: Get out of their way. At this point, you need to make it easier for them to get their jobs done. Most software development processes are about making it harder to do the wrong thing, which inevitably makes it harder to do the right thing. For many managers, software processes are like violence: "If some isn't good enough, more must work better." Don't fall into this trap.

Various things that will help (and not hinder) your developers:

• Provide direct access to stakeholders with requirements, including customers if possible, domain experts, etc.
• Fire poor performers quickly. Sooner if possible. Let the team decide who these people are (you'll hear the complaints quickly if you are receptive).
• Get the team to agree on formatting and design conventions. This will save you more time and frustration than you know. People who insist on their own conventions are unprofessional and should have failed the criteria for being hired (they're not good team members).
• Identify the team tie-breaker. You may use the title "team lead", "project manager", "architect", whatever. This is the person who makes the call when consensus doesn't happen quickly.
• Get an effective issue tracking system. Bugzilla is a free and minimal option. Scarab may be ready for prime time (but wasn't a year ago when I last checked). Trac is simple but clean (and once they add the issue process enhancements... woo hoo). Rally is commercial and hosted, but very effective for agile teams.
• Install subversion (use the file-based installation, the db installation option is still flaky). Understand and utilize subversion's features (branches and variations on branches are #1 on this list).
• Protect the tip with a minimum of a nightly build. There are build checkers (calavista is good, there are some open source alternatives that can frequently check the status of the tip... I wish I could remember some names).

When you get to this point, you'll have read a lot more about software development (your good developers can recommend some fantastic books) and you'll have much more precise questions.

If you don't read at least "The Mythical Man Month" [amazon.com] by Fred Brooks before managing a software team, you will fail and you won't understand why.

Regards,
Ross