Well, thanks for the opinion. But FWIW most of
the SPEC people I actually work with every day
believe that SPEC really does add "an ounce of
honest data" (which is worth more than a pound
of marketing hype). See SPEC's
background/philosophy statement.
Yes, even honest data can be misunderstood, taken out of context, or abused.
Example misunderstanding: the comment that said
that compiler optimizations are "cheating". Nope,
compiler optimizations are allowed by design. The
SPEC CPU benchmarks are specifically intended to
test CPU, memory hierarchy (caches & main memory),
and compilers.
(Of course, compiler optimizations are expected to be generally useful, and not unfairly "target"
a specific benchmark. That's a judgment call which SPEC has sometimes had to work hard to make.)
Now, one can argue about whether Apple's attempt to hold the compiler constant was a reasonable thing to do, given that the CPU2000 benchmarks are intended to test all three of cpu/memory/compiler. But they at least have told us what they did, and how they did it, and in my book that goes a long way toward supplying the ounce of honest data.
For those who suggest that improvements should be made to the SPEC benchmarks in one way or another, please be reminded that you can join SPEC (discounts for academic/non-profit memberships) and you can contribute to new benchmarks (and earn modest financial compensation if your benchmark is accepted.)
SPEC has a long history of welcoming technical contributions by technical people (but is less responsive to complaints - we know it's impossible to please everyone).
Disclaimers: I am not employed by Apple, nor do I own an Apple computer. I am employed by another company that is a member of SPEC. My opinions are my own and have not been approved or disapproved by my employer, by SPEC, or by Apple.
-john henning
