QA Consultants Determines AMD's Most Stable Graphics Drivers in the Industry

[bug]

If you read these results, you'd think driver stability was actually a big problem, and it really isn't. It makes you think crashes are happening constantly, and I can say, using cards from both camps on a daily basis, that the reality is crashing is really uncommon. I can't even remember the last crash I had, that's how long ago it happened.

And stability is only part of the picture of a good driver, annoying bugs and how quickly they are fixed, how often they are updated(it can be too often as well as not often enough), etc. And again, as someone that uses drivers from both sides daily, I'm going to say right now they are both about even. I can not honestly say I prefer one over the other as they are today.

Well, they ran one specific test out of one specific suite. You just can draw a generic conclusion from that.
On top of that, the test they picked doesn't seem to be marked as very relevant by Microsoft: "To qualify for the Windows Hardware Compatibility Program, your product must pass certain tests using the Windows HLK." All the tested cards qualify (they have WHQL drivers), but all of them fail that test from time to time. So this test isn't one of the tests needed to qualify.

Long story short: "independent" test between two competitors, paid by one of them, looking at one specific aspect - the very definition of throwaway data.

 

[B-Real]

Since Crimson and Adrenalin, AMD developed the overall quality of their drivers meanwhile NV just got worse and worse. I don't say that AMD drivers are better now, but at least are on par with the green drivers. Which is nice considering they have less resources. And, as someone wrote earlier, NV releasing hotfixes of hotfixes is pretty obvious. This was typical in Catalyst times.

 

[arbiter]

This picture is pretty telling:

The Radeon cards are being held back by the memory subsystem. In tests where memory performance isn't the bottleneck, the Radeon cards do well. In instances where it is, they perform poorly.

Still, this isn't my point. There's an open standard out there for compute and NVIDIA deliberately doesn't update it because they would rather promote their proprietary solution (just like G-SYNC).


OpenCL 2.0 features a new shared memory subsystem that vastly accelerates memory accesses:
https://www.anandtech.com/show/7161...pengl-44-opencl-20-opencl-12-spir-announced/3
I wouldn't be surprised if AMD jumped on OpenCL 2.0 for that reason and NVIDIA are dragging their heels because it makes CUDA look bad.

Not seeing any benchmarks that compare OpenCL 1.2 and OpenCL 2.0 performance.

Thing with the idea that Memory subsystem being bottleneck kinda weak over all when you look at things and shows AMD needs to put work in it. Vega 64 has an effective memory bandwidth of 483GB/s, While GTX1070 only 256GB/s. Almost half the bandwidth yet beats it.

 

[bug]

Thing with the idea that Memory subsystem being bottleneck kinda weak over all when you look at things and shows AMD needs to put work in it. Vega 64 has an effective memory bandwidth of 483GB/s, While GTX1070 only 256GB/s. Almost half the bandwidth yet beats it.

I think he meant something is bottlenecking access to all that bandwidth. We all know that AMD's highly parallel architecture has been pretty much impossible to feed, so that's always a possibility.

 

[FordGT90Concept]

I don't know why but Vega 64 shouldn't be that low and that's a problem. Even GTX 1060 spanking the 580 which has a bus 25% wider makes no sense. Either AMD failed at memory or the test is completely flawed. Considering they tested OpenCL 2.0 on AMD, OpenCL 1.2 on NVIDIA, and were using 2 year old builds of the benchmark at the time of testing, I'd say the test is flawed. Yes, AMD has a memory subsystem issue as bug pointed out but it isn't that much of an issue. There's more afoot.

Case in point: OpenCL Ethereum mining (where the developers have a vested interest in making it run as fast as it can on all platforms), AMD runs away as it should.

 

[toyo]

This one of the most laughable "studies" I've seen. AMD bought, AMD published, comments on video closed, likes/dislikes closed, AMD sent in the cards - how were nvidia cards obtained? cheap motherboard, cheap PSU used, Quadro and WX cards failing more than a Vega, the 1060 and 1050 suddenly being far more unstable than a 1080ti, and it goes on an on.

Desperate move that was not needed.

 

[FordGT90Concept]

AMD buys NVIDIA cards and NVIDIA buys AMD cards to check if problems are isolated to their own hardware, also to benchmark their cards against competitors. AMD sent a selection of cards in their possession (AMD and NVIDIA) to QA Consultants for testing.

Testing was performed using 12 identical systems, 6 tests were ran, cards were swapped between systems, and 6 more tests were ran. If the problem was the hardware, the failures should have been consistent.

There is something wonky going on with Quadro and FirePro. That's kind of the point of doing these tests: issues like that can be found and worked out.

 

[efikkan]

Well, at least I do appreciate the consultants providing some documentation of their findings:

We define graphics driver stability as a resistance to Blue Screens (BSODs), application crashes, hangs, and otherwise unexpected behavior in the presence of stressful test vectors over a period of time. … CRASH is a GPU stress test tool that spans 4 hours in length and captures test cases covering S3, display resolution changes, display orientation changes, content protection, and rendering.

Unfortunately the 125 page report tells us nothing about which specific tests failed.

I would have broken down the testing into specific categories:
- Desktop usage (resolution changes, etc. like their test covered)
- Video decoding
- API conformity
- Rendering quality (under stress testing)
- Frame pacing/stutter
(- Multi-GPU?)

-----

I see many of these tests show more failures for specific GPUs than others. If e.g. a GTX 1060 fails a lot while GTX 1080 Ti and GTX 1050 does not, then it's more likely a bad sample than a driver issue. But it's easy to eliminate such variation; just increase the sample size in the test, have 10 of each card, if all of the GTX 1060 show the same symptoms then you actually got something, except this data which is just statistically useless.

So unfortunately, this test just isn't good enough.

 

[bug]

So unfortunately, this test just isn't good enough.

Not enough for what?
If the goal is to smear the competition, a paid for test will do just fine (remember, nobody had a problem with GPP till AMD "brought" it to Kyle's attention).
If the goal is to learn something from it, it does fall short, as you have noted.

 

[R-T-B]

Myth busted...

Commissioned by AMD

From the article. I wouldn't put much credence in this.

 

[Xzibit]

AMD buys NVIDIA cards and NVIDIA buys AMD cards to check if problems are isolated to their own hardware, also to benchmark their cards against competitors. AMD sent a selection of cards in their possession (AMD and NVIDIA) to QA Consultants for testing.

Testing was performed using 12 identical systems, 6 tests were ran, cards were swapped between systems, and 6 more tests were ran. If the problem was the hardware, the failures should have been consistent.

There is something wonky going on with Quadro and FirePro. That's kind of the point of doing these tests: issues like that can be found and worked out.

It brings into question what exactly is the baseline or threshold of which a product gets certified. Does it only have to pass the test once and to what extent.

Digging into MS HLK you can still have a failing product get certification with a contingency (Agreeing to fix the problem with-in a agreed time frame)

 

[FordGT90Concept]

Does it only have to pass the test once and to what extent.

Yes. AFAIK, to get WHQL certified, only has to pass the test once on a single card. Because AMD and NVIDIA separate their professional and non-professional drivers, they each have one test each on a professional and non-professional card:
Radeon driver -> Radeon card
Radeon Pro driver -> Radeon Pro card
GeForce driver -> GeForce card
Quadro driver -> Quadro card

In this test, each card was subjected to the same test 6 times per day for 12 days (effectively running the benchmark 72 times per card in total).


Edit: Looking at results again, it's pretty clear that AMD is doing WHQL testing on Radeon Pro WX 7100 (Polaris 10), not Radeon Pro WX 9100 (Vega) and definitely not Radeon Pro WX 3100 (probably Polaris 11). NVIDIA is likely doing Quadro testing on a Volta card (Quadro GV100, not represented in the test) which is why there are failures on Quadro P600 (Pascal), Quadro P4000 (Pascal), Quadro P6000 (Pascal).

Digging into MS HLK you can still have a failing product get certification with a contingency (Agreeing to fix the problem with-in a agreed time frame)

Yeah, because known issues for one specific hardware configuration shouldn't hold back a driver release for all of the other ones.

 

[arbiter]

AMD buys NVIDIA cards and NVIDIA buys AMD cards to check if problems are isolated to their own hardware, also to benchmark their cards against competitors. AMD sent a selection of cards in their possession (AMD and NVIDIA) to QA Consultants for testing.

If AMD also provided the nvidia cards to test against that would cast even been cloud over these test's as that would open the door for possibility those cards were picked on purpose cause they were known to be failures. Not saying it did happen but opens the door for the idea. To make the test fair Nvidia should been allowed to provide the cards from their side but no indication that happened or if they even gave nvidia the chance to. They could of got them from retail channels but we don't know as not info provided on it.

 

[bug]

If AMD also provided the nvidia cards to test against that would cast even been cloud over these test's as that would open the door for possibility those cards were picked on purpose cause they were known to be failures. Not saying it did happen but opens the door for the idea. To make the test fair Nvidia should been allowed to provide the cards from their side but no indication that happened or if they even gave nvidia the chance to. They could of got them from retail channels but we don't know as not info provided on it.

Tbh the fair thing to do would be for the tester to source the cards themselves. However, I do not believe AMD intentionally send faulty cards. My only issue is that we're given a very narrow view of the tests performed, thus we cannot infer anything useful from them.
Like @efikkan pointed out above, it's one thing to crash/render incorrectly under heavy (even unrealistic) load, yet it's entirely another thing if the card doesn't handle 100 resolution changes per minute.

 

[Xzibit]

Tbh the fair thing to do would be for the tester to source the cards themselves. However, I do not believe AMD intentionally send faulty cards. My only issue is that we're given a very narrow view of the tests performed, thus we cannot infer anything useful from them.
Like @efikkan pointed out above, it's one thing to crash/render incorrectly under heavy (even unrealistic) load, yet it's entirely another thing if the card doesn't handle 100 resolution changes per minute.

Might want to read the PDF again.

Within Windows HLK, we used the 64-bit variant of CRASH to stress the Graphics Processing Unit
(GPU). CRASH is a GPU stress test tool that spans 4 hours in length and captures test cases covering
S3, display resolution changes, display orientation changes, content protection, and rendering.

 

[bug]

Might want to read the PDF again.

It doesn't say what failed in each instance though.

 

[FordGT90Concept]

A WHQL'd driver should never fail, that's kind of the point.

The Quadros were all PNY which is what most Quadros are.

 

[bug]

A WHQL'd driver should never fail, that's kind of the point.

The Quadros were all PNY which is what most Quadros are.

That is obviously not the point as all drivers fail and all drivers are WHQLed. That's why I'm saying these tests raise more questions than they answer.

Anyway, discussions are mostly moot at this point. For years to come, when driver stability will be questioned, this "study" will be brought up as an argument. Mission accomplished AMD.

 

[FordGT90Concept]

That is obviously not the point as all drivers fail and all drivers are WHQLed.

When was the last time your NIC driver failed? How about the ACPI Multiprocessor driver? What about the USB host driver? Display drivers are quite unique in their rate of failure and they should not fail as they do.

Beta drivers are not WHQL'd.



As First Strike said:

The configuration of a production environment is strictly controlled in every corporation. Configurations must be intensely tested prior to installation, and tweaked, if necessary. After a configuration passed such tests, this is where "stability" kicks in -- is such success steadily reproducible in the following runs?

PNY Quadro has a reputation for having issues. Businesses test before deployment and strictly control what drivers they use as a learned behavior to counter that unreliability. AMD and NVIDIA both have serious problems with driver stability on their professional cards. It begs the questions: why are they separate at all? Are AMD and NVIDIA mostly selling a bridge to no where for a massive premium?

 

[bug]

When was the last time your NIC driver failed? How about the ACPI Multiprocessor driver? What about the USB host driver? Display drivers are quite unique in their rate of failure and they should not fail as they do.

Beta drivers are not WHQL'd.

What beta drivers? Neither GeForce 397.64 nor Adrenalin 18.5.1 are beta drivers? Wth are you talking about?
They're both WHQLed and failing these tests. Hence, passing all tests is not a requirement for WHQL certification.

 

[FordGT90Concept]

Hotfixes, usually. For example, 16.7.2:
https://drivers.softpedia.com/blog/...6-7-2-hotfix-is-out-download-now-506175.shtml

Here's the download URL, note the "non WHQL" in there:
https://www2.ati.com/drivers/beta/non-whql-64bit-radeon-software-crimson-16.7.2-win10-win8.1-win7-july9.exe

AMD later updated the package with WHQL:
https://support.amd.com/en-us/kb-ar...are-crimson-edition-16.7.2-release-notes.aspx

Important Update
-These drivers are now Microsoft WHQL certified

 

[bug]

Ok, you totally lost me. How is that relevant to the discussion we were having?

 

[HD64G]

From the article. I wouldn't put much credence in this.

Well, my experience with AMD drivers for the past 5 years is that they are good and getting better by the month. So, myth busted for sure for whoever pretended that this is not the case. And for any research someone interested pays always. If what you say is truth, not any research in our history should be trusted.

 

[FordGT90Concept]

Ok, you totally lost me. How is that relevant to the discussion we were having?

Yeah, I misunderstood and got derailed.

I think the moral of the story is that WHQL testing needs expanding because just running the 4 hour test once isn't enough to catch a lot of problems.

 

[bug]

Well, my experience with AMD drivers for the past 5 years is that they are good and getting better by the month. So, myth busted for sure for whoever pretended that this is not the case. And for any research someone interested pays always. If what you say is truth, not any research in our history should be trusted.

Tbh, I think driver stability hasn't been much of an issue since drivers got TDR and Microsoft pushed WDDM2.0 out (hell, even WDDM 1.x was pretty stable, but 1.0 surely rubbed a lot of manufacturers the wrong way). Complaints about drivers have been generally geared towards added overhead in some areas or control panels being resource hungry, unresponsive or getting seemingly random changes at times.