Big Challenges for Vista Bug Hunters 213
The New York Times is reporting on the final rush to bug fix Windows Vista. Even with massive numbers of testers and five years of work behind them, the folks in Redmond are pushing it to the wire in order to make sure it releases soon. From the article: "Vista has also been tested extensively. More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft. Such data supplements the company's own testing in a center for Office referred to as the Big Button Room, for the array of switches, lights and other apparatus that fill the space. (A similar Vista room has a less interesting name -- Windows Test Technologies.) This is where special software automatically exercises programs rapidly while looking for errors."
Re:I wouldn't want to be the guy (Score:3, Interesting)
The real world is probably worse than the statistics suggest.
I tried to install Vista on three PCs, all of which passed Microsoft's Ready for Vista testing tool, but all three failed before they were able to sent any crash data back to Microsoft. Two installs hung due because Vista didn't like my SATA / motherboard combination. The other got its knickers twisted over my partitioning scheme. And that was before I got a chance to find out if any of my other hardware (printer, scanner, TV card, firewire, network, graphics, CD/DVD, monitor, soundcard, suspend/resume, camera, etc) had any kind of working support.
Recent Linux kernels work perfectly with all this hardware. Could Vista be the first Microsoft OS that lags behind Linux for hardware support?
Yes, you can use hardware to track down bugs... (Score:5, Interesting)
With XP, almost all of the crashes are due to bad (usually non-MS) device drivers. If you run a system with pure MS drivers and quality hardware you'll never see a BSOD. If you run the usual business suite of software (Office, Outlook, IE) you probably never see an application crash.
It's the crappy hardware and badly written drivers that cause the crashes. That's the difference with Apple.... since they control the hardware there's less crashes due to bad hardware and there are fewer third-party drivers for Mac hardware. The software is probably the same quality.
Why try, and fail to, reinvent the wheel... (Score:3, Interesting)
Microsoft wanted a more reliable machine, improved memory management, a better filesystem, etc... Instead of throwing resources at doing these things from scratch, why didn't they just
From every point of view it seems to make more sense. They spend less money, get a more reliable product that can run very nicely on existing hardware, get some good press for a change, and benefit from the work of unpaid open-source programmers all over the world. But it isn't their way.
What about compatibility? (Score:3, Interesting)
Something that I haven't heard much recently is about Vista compatibility. MS has said before that it will be compatible and for most software and hardware, it was true in previous versions. But there were enough exceptions. ME was supposed to be backwards compatible. But many specialized drivers had to be written for it. XP definitely required some driver updates. Since Vista is a architectural change, so one would except some compatibility issues especially when DRM and enhanced security is thrown into the mix.
Technically would MS classify incompatibility as not a a bug, especially if is does not cause a crash?
Big bang testing (Score:3, Interesting)
(Yes I am aware I used singular, it was a joke, OK?)
Re:special software (Score:1, Interesting)
The automation system is responsible for running tests autonomously, which includes imaging machines, installing the product, distributing tests to machines, ensuring that a test-gone-bad does not prevent further tess from running, analyzing and storing results, etc. The database backing the automation system has roughly 50gb of data (consisting primarily of scenario and results data).
"Big Button" is actually a part of the test automation system which allows developers to create a change on their machine and run a batch of tests against it in a test lab. It handles collecting the changed bits, distributing them to various machines, selecting tests to run against the change, etc. One of its more notable features is that it allows developers or testers to specify failures to hold machines on in order to more readily reproduce, investigate, and fix intermittent failures.
During any given day, this system will run roughly 100,000 test scenarios against various builds of the product.
You laugh and joke about how it must be some archane system held together with shoestrings and bubblegum, but it is by far and wide the most advanced system of its type I've ever seen or heard of.