Screaming Penguin - Source Does Matter

A pretty convincing case has been building up against Nvidia in terms of the FutureMark 3DMark benchmark wars. NVidia appears to have told its card to not do the work in certain cases and just tell 3DMark it did do the work in order to get higher scores. Pretty ridiculous, but I guess it just reinforces caveat emptor. After all NVidia does not own FutureMark, they just did what they knew would help with a popular third party test utility. Its bad, its petty, its silly, but its not some egregious affront to humanity as some have taken it.

For more see the linked Ars article.

This comment on Ars sums up what they did wrong:

Second off, if you read the paper, you'll find out that there were basically two things that NVIDIA did:

1) Ignore the back buffer clear instructions at certain points, because they don't affect the image quality. This seems like an optimization, but it isn't: the driver can't a priori know when it won't affect the image unless the coder knows the exact execution path. You could NEVER do this in any interactive demo.

2) (this one's completely unforgivable) Completely ignore certain shader programs and substitute them for ones which do similar, but not exactly the same effects. This is exactly what ATI did with Quake 3: reduce image quality for speed. Yes, the shader paths are not as fast as they could be, but the program wanted high image quality, not crappy ones.

This IS a cheat. Flat out. NVIDIA realized that everyone stopped using in-game FPS counts to benchmark games, and instead, they're using timedemo and flybys with repeatable known instruction paths, and in order to gain a temporary boost to their scores, they took advantage of that. It's a limitation in the way we do ALL benchmarks, and someone should look to see if this happens in other games as well. Somehow I doubt NVIDIA's alone in this too: it's an obvious flaw in the design of all common benchmarks.
  Nvidia caught cheating with 3DMark2K3