1. As a Software Developer, it is hard
2. Nvidia won't be able to do it because they don't know how each developer has designed their games (as in which files and how to benchmark).
3. Games developers won't do it because it would be a waste of their time and money and if each game has its own benchmark tool, it will be redundant; think of HD space waste.
+1 for truth on this.
The fustrating part about all of this is that, with QA for the most part being outsourced, trying to find ANY external testing company who REALLY knows their stuff regarding gaming and PC hardware compatibility is incredibly difficult. Most will fall into one of two camps:
- Unit testing: Where a "unit", in this case a desktop PC either OEM or hand-built, is used amongst a pool of testers. They then work down a "checklist", most commonly known as a test case, to use during their testing cycle. Typically this the most commonly found kind of hardware compatibility especially in those companies who outsource their QA testing.
- Benchmark testing: This is the lesser known, and based upon my experience dying, example of hardware compatibility testing. This kind of testing is very similar to what hardware reviewers use with a few key differences:
a. It is geared more towards testing the individual hardware components than the overall configuration
b. The tester will have a low (read: min spec), mainstream, and recommended set of configurations. A typical mix of this will have a Intel/NVIDIA "set" of systems and AMD/ATI for the other
c. That, based upon the experience of the tester, will mix and match some configurations to ensure that sticking an NVIDIA video card in an AMD motherboard will not cause a game to choke
The bigger and more well financed QA groups will have an internal team, with a hardware catalog to test with, but sadly this is slowly dying. This is somewhat depressing when I recall having a conversation with a QA outsource vendor the company I was with at the time had a difficult time wrapping their head around identifying performance-related issues and instead just provided "test case results" that centered around the failed "unit" that they were using and left it up to the developer to try and figure out what the issue was.
Sadly, the expertise involved with PC hardware testing is not sticking around the industry due to either being laid off in preference to outsource testing (with EA being one of the bigger companies that did away with a good portion of their internal testing teams from what I understand) or the persons with the requesite knowledge not wishing to work the long hours and poor financial benefits to keep the knowledge in the industry.