Monday, July 16th 2018
QA Consultants Determines AMD's Most Stable Graphics Drivers in the Industry
As independent third-party experts in the software quality assurance and testing industry for over 20 years, QA Consultants has conducted over 5,000+ mission-critical projects and has extensive testing experience and depth in various industries. Based in Toronto, Ontario, QA Consultants is the largest on-shore software quality assurance company, with a 30,000 sq. ft., industry-grade facility called The Test Factory.
Commissioned by AMD, QA Consultants used Microsoft's Windows Hardware Lab Kit (HLK) to conduct a 12-day head-to-head test comparing AMD and Nvidia graphics drivers. The test compared 12 GPU's, six from AMD and six from NVIDIA, running on 12 identical machines. All machines were configured with Windows 10 April 2018 Update. Both gaming and professional GPUs were equally represented in the testing. After running for 12 days of 24-hour stress tests, the aggregate of AMD products passed 93% of the scheduled tests. The aggregate of NVIDIA products passed 82% of the scheduled tests. Based on the results of testing the 12 GPUs, QA Consultants concludes that AMD has the most stable graphics driver in the industry.
Sources:
QA Consultants, Graphics Driver Stability Report
About QA Consultants
QA Consultants is the North American leader in software quality assurance and testing services. Having successfully delivered over 5,000 testing and consulting projects to a variety of sectors including automotive and transportation, advertising and marketing, banking and finance, construction, media and entertainment, US & Canadian Federal State and Local government, healthcare, insurance, retail, hospitality, and telecommunications.
The Test Factory is the next generation of software testing, providing a superior quality, cost and service alternative to offshore providers and external contractors. The centre can handle any testing project of any size, with any application and for any industry. With full-time employees in Toronto, Ottawa and Dallas, QA Consultants supports customers by providing testing services such as accessibility testing, agile testing, test automation, data testing, functional testing, integration testing, mobility testing, performance testing, and security testing. Along with engagement models like Managed Consulting Services and On Demand Testing , QA Consultants is equipped to handle any client's request.
124 Comments on QA Consultants Determines AMD's Most Stable Graphics Drivers in the Industry
Case in point: OpenCL Ethereum mining (where the developers have a vested interest in making it run as fast as it can on all platforms), AMD runs away as it should.
Desperate move that was not needed.
Testing was performed using 12 identical systems, 6 tests were ran, cards were swapped between systems, and 6 more tests were ran. If the problem was the hardware, the failures should have been consistent.
There is something wonky going on with Quadro and FirePro. That's kind of the point of doing these tests: issues like that can be found and worked out.
I would have broken down the testing into specific categories:
- Desktop usage (resolution changes, etc. like their test covered)
- Video decoding
- API conformity
- Rendering quality (under stress testing)
- Frame pacing/stutter
(- Multi-GPU?)
-----
I see many of these tests show more failures for specific GPUs than others. If e.g. a GTX 1060 fails a lot while GTX 1080 Ti and GTX 1050 does not, then it's more likely a bad sample than a driver issue. But it's easy to eliminate such variation; just increase the sample size in the test, have 10 of each card, if all of the GTX 1060 show the same symptoms then you actually got something, except this data which is just statistically useless.
So unfortunately, this test just isn't good enough.
If the goal is to smear the competition, a paid for test will do just fine (remember, nobody had a problem with GPP till AMD "brought" it to Kyle's attention).
If the goal is to learn something from it, it does fall short, as you have noted.
Digging into MS HLK you can still have a failing product get certification with a contingency (Agreeing to fix the problem with-in a agreed time frame)
Radeon driver -> Radeon card
Radeon Pro driver -> Radeon Pro card
GeForce driver -> GeForce card
Quadro driver -> Quadro card
In this test, each card was subjected to the same test 6 times per day for 12 days (effectively running the benchmark 72 times per card in total).
Edit: Looking at results again, it's pretty clear that AMD is doing WHQL testing on Radeon Pro WX 7100 (Polaris 10), not Radeon Pro WX 9100 (Vega) and definitely not Radeon Pro WX 3100 (probably Polaris 11). NVIDIA is likely doing Quadro testing on a Volta card (Quadro GV100, not represented in the test) which is why there are failures on Quadro P600 (Pascal), Quadro P4000 (Pascal), Quadro P6000 (Pascal).Yeah, because known issues for one specific hardware configuration shouldn't hold back a driver release for all of the other ones.
Like @efikkan pointed out above, it's one thing to crash/render incorrectly under heavy (even unrealistic) load, yet it's entirely another thing if the card doesn't handle 100 resolution changes per minute.
The Quadros were all PNY which is what most Quadros are.
Anyway, discussions are mostly moot at this point. For years to come, when driver stability will be questioned, this "study" will be brought up as an argument. Mission accomplished AMD.
Beta drivers are not WHQL'd.
As First Strike said:PNY Quadro has a reputation for having issues. Businesses test before deployment and strictly control what drivers they use as a learned behavior to counter that unreliability. AMD and NVIDIA both have serious problems with driver stability on their professional cards. It begs the questions: why are they separate at all? Are AMD and NVIDIA mostly selling a bridge to no where for a massive premium?
They're both WHQLed and failing these tests. Hence, passing all tests is not a requirement for WHQL certification.
drivers.softpedia.com/blog/amd-crimson-edition-16-7-2-hotfix-is-out-download-now-506175.shtml
Here's the download URL, note the "non WHQL" in there:
https://www2.ati.com/drivers/beta/non-whql-64bit-radeon-software-crimson-16.7.2-win10-win8.1-win7-july9.exe
AMD later updated the package with WHQL:
support.amd.com/en-us/kb-articles/pages/radeon-software-crimson-edition-16.7.2-release-notes.aspx
I think the moral of the story is that WHQL testing needs expanding because just running the 4 hour test once isn't enough to catch a lot of problems.