Non-3rd party benchmarking must be ignored (Score 1)

Any benchmark run by an interested party has to be ignored as biased. I've been on the inside of graphics h/w companies (engineering, both UNIX and PC-based) for 10 years, and have seen time and time again the games vendors play to make themselves look good. The schemes used run from simple to very complex. The simplest method is just to not do apples/apples: manipulate/choose the tests so that the competition goes down the slow path. The more complex ones involve what most people would call "cheating" to increase your scores. Someone already mentioned dropping triangles as one such cheat. Dropping tris that don't light pixels is fine, since they don't change anything, but dropping triangles that light pixels doesn't produce the right picture and is a cheat. The thing is that it's *hard* to determine whether or not a small area triangle (other than zero area) actually lights a pixel, so vendors who do this cheat usually just say that all triangles with area below a particular threshold value are culled. There are infinitely many ways to cheat, and any benchmark, to be taken seriously, must be run by a 3rd, uninterested party, and whatever platform you're running on should have conformance/correction/cheat detection tests. This task becomes even harder when you consider that multiple platforms are involved here.

In conclusion, all that can be gleaned from this benchmark (assuming no intentional cheating is involved) is that the particular versions of the software used on the particular platforms at the time of the testing had the resulting performance. Unless a rigorous and lengthy evaluation process was used to insure that ALL the same level of acceleration was running on ALL the same platforms, these results in terms of comparing the "multimedia performance" of these systems is meaningless.