Accurate Methods For Benchmarking Hardware?

General Breetai asks: "This is yet another benchmark showing DDR is faster than RDRAM. I'm not posting this as another DDR vs. RDRAM debate, but rather to question the benchmark methods. When looking through the results, it's clear that the OS and test SW has a huge impact on the results. How can we trust SW benchmarks to test a particular subsystem when it's clear that the SW relies on the entire system? In the case of new memory systems, I propose that the only reliable method is to compare published timing data in a theoretical setting. SW is just too dependent on too many things to be reliable, especially if it is running on top of a complex OS." Are there ways to benchmark hardware components using some measure that isn't explicitly tied to the OS on which said benchmarks run?

"Before all of you SW types out there get all defensive. You'll have to prove to me how the benchmark software is not affected by the entire system. Even between to identically configured systems, there will still be some amount of random error incurred by the OS's management of the system. On top of that, when trying to evaluate a particular subsystem's performance, you cannot show how other subsystems are affecting that performance. For instance, when moving large chunks of data in and out of memory to measure memory performance, how can you prove that no other part of the system is stealing bandwidth? How do you know that the AGP or HDD drivers aren't moving data as well? In addition, since a lot of performance differences are controlled by low level OS drivers and BIOS configurations, how do you know that you are truly measuring the memory performance, and not how well the OS drivers are optimized? BIOS can also play a big part. There have been many cases where performance increases with a new BIOS version because better chipset settings where used. This can be viewed best when two boards with identical chipsets are compared, and one performs better than the other, sometimes significantly. For the above mentioned article, one can ask if the results are skewed by the use of custom Micron OS improvements required to support their Samurai chipset? The conclusion, IMHO, is that benchmarks should be viewed as overall system measurments, even if they are designed to measure a specific part of the system.

The best way to measure a hardware subsystem like the memory channel, is to sit down a simulate transactions based on the published electrical timing specs provided by the vendors. Since these memory busses have very specific protocols, you can critically evaluate the performance based on how well the protocol performs under specific theoretical circumstances. I.E choose a specific amount of data to transfer, and then show how well the two different protocols can move the data. You can then tweak the parameters to find the best settings. In doing so, you can show, down to the picosecond, which protocol can move data the fastest without interference from the rest of the system. However, this all has to be done in a theoretical situation, independent of OS & SW.

By adding the rest of the overall system and the OS which can both steal uncontrolled bandwidth, you are actually yeilding suspect results. Let the flames fly, but what are your thoughts on this? Has anyone seen a hardware/electrical timing comparison between RDRAM and DDR?"

But aren't we interested in real world results?

[crisco]

OK, by benchmarking by the electrical specs we find out which type of hardware COULD be faster, but really, we're benchmarking because we're interested in real-world performance. We're interested in the quirky behaviour the rest of the hardware and the operating system introduces. Thats why application based benchmarks gain some degree of respect in benchmarks such as SPEC or TPC.

Obviously, the most useful benchmarks would then provide detailed information on the entire configuration, possibly contrasting various configurations to point out slight differences that arise because of hardware, operating system and the like.

In the case you mention, I'd be extremely interested in knowing how DDR and Rambus work with different operating systems or different motherboards. And I'd be interested in seeing the differences with different types of applications. Most useful would be details on WHY the differences arise. Of course, that brings us full circle, back to the hardware details that you are interested in.

Best way to benchmark . . .

[KevF]

Is to use a soft bench and note the damge made to it by dropping hardware on it from different heights.

Re:But aren't we interested in real world results?

[Chris Pimlott]

I have to agree with the above post. It reminds me of a recent thread on linux-kernel as related in Kernel Traffic #80 [linuxcare.com].

Basically, Andy Kleen made a patch he believed would slightly improve performance and posted some bechmarks he came up with. Linus asked for some real benchmarks in a user mode situation. Andy replied

Doing it completely from user space would probably add so many other variables and variances that the results would be hard to interpret,

to which Linus countered

The other way to say the same thing is

"Doing it from user space might show that it's not a performance optimization that can be noticed".

For better or for worse, the real world is the one we live in.

Benchmark OS

[cronio]

I'd think that the "best" way to benchmark would be to have a non-multitasking kernel that can fit on a disk that, when booted, would run the benchmark(s), display the results, and quit. This would get rid of most of the abstraction between the benchmark program and the hardware, giving the best possible results for the hardware.


Not reading .sig

Could it be the..

[L+st]

..6th post?