Anyone with any brain knows that in order to determine with any degree of certainty you need to test under real-world conditions. Real-world conditions include things like identifying beta users in several departments of a large company and allowing them to run beta software in order to test for the masses. Testing under virtualization is just smoke testing - verifying that your users don't have any serious difficulties with the new OS. Vista has a "burn in time" - from initial installation to some level of full capability. The file indexing database, for example, is a serious piece of technology which requires days upon days of use in order to reach a steady state. As operating systems become more complex and include more heuristic features, testers who take the attitude of just virtualizing and following a script will find themselves with more of a support headache than testers who opted to not only script test, but also put the beta product in real-world situations and find any issues.

Real-world testing uncovers many more issues (and more complex issues) than virtual script testing can ever hope to. This is, in fact, one reason that Microsoft chooses to release betas to the public.

In testing a beta product, a certain amount of resource overhead should be calculated in order to effectively allow for testing. For something as wide-ranging as an OS release, 10% of a beta user's time would need to be allocated to work with tech teams in diagnosing, reporting, and working around problems that are uncovered. Once the beta period is complete, I expect users to be able to upgraded to production with a minimal hassle. During SP1, we had users that were down for more than a day - not to mention loss of productivity post re-installation due to the OS's burn-in time.

I won't say that this approach doesn't have risk. Of course the action of placing real-world users on a beta product has profound implications especially with respect to an OS product.

Some ways to reduce risks associated with users running beta software under real-world conditions are:

- Smoke-testing in virtualized systems prior to even allowing the beta to continue. Pay attention to "show stoppers" AND their potential aggregate effects. Don't ignore issues because you think you have a work-around.

- Build contingencies for disastrous events (such as backing up or replicating offside more aggressively than for typical users).

- Have a reversion plan tested and in-place prior to beta go-live.

... and here's where we went wrong with SP1: Don't believe the vendor when they tell you there will be an upgrade path from SP1 Beta to SP1.

I, for one, will NOT be installing any beta service packs from Microsoft and I'll be recommending my company do the same. If you were unfortunate to test the SP1 beta, you'll recall that you were forced to re-front your machines after the beta period was over. Until Microsoft guarantees that it will provide a reasonable upgrade path from the beta to production, there is no point in testing until this becomes public.