Is CPU game benchmarking methodology (currently) flawed?

[HTC]

I don't think so, here comes the answer:

You guys were simply too fast. This guy is smarter than you thought he is and he didn't really do a mistake, I guess people simply didn't understand his logic. It's a fact, 8350 is now faster than at release, it's probably faster than 2500K when properly utilized. But then again, this is just vs. a 2500K at *stock* which means nothing. Who bought a K CPU, which cost extra, to not overclock it? Yeah. That said, the FX 8350 has not the slightest chance once the 2500K is overclocked, even slightly. The difference just increases with a higher clock. In the end, all that information was just good for one thing: was the FX 8350 a futuristic CPU ahead of its time? Yes. Is it better now than at release? Probably, but it's still not good enough. A overclocked 2500K is barely good enough, but a FX 8350 isn't.

I do admit that i missed the very important bit that different games being tested was the point. I'm also to blame in the sense that i took HUB's numbers as a "gospel" (like Adored calls it) in order to "prove" that the methodology wasn't flawed after all.

His more recent video shows that current CPU benchmarking indeed is flawed (in this title @ the very least). Let me however say that what i'm trying to show is not Intel VS AMD CPU performance bit but the difference you get in a supposedly CPU bottlenecked game when changing from a nVidia card to an AMD one on BOTH CPUs:

You can't have it both ways: either the CPU is being bottlenecked or it isn't. Adored showed that both Intel and AMD benefited from changing from nVidia to AMD in DX12 and both lost in DX11. That the gap shrunk in DX12 is not the issue i'm trying to address: that there is a gap is the issue i'm trying to address. This proved that the CPU wasn't being bottlenecked after all, or there wouldn't have been an increase in both CPUs: there was another variable that wasn't being accounted for.

But there are variables here, because of which i think more testing is definitely required: he tested crossfire VS single card and that introduces another variable that doesn't have to be present: CF scaling. It seems RotTR isn't the only game because it happens in DX12's The Division, that i know of so far.

In fact, you don't even need to use 2 CPUs to test if this is true or not, but you do need an nVidia and an AMD cards, as well as RotTR game: just run RotTR using DX11 and DX12 with settings you're absolutely sure will bottleneck the CPU with both cards @ stock and then with the highest overclock on the cards you can get.

If the CPU is bottlenecked "properly", then going from a stock nVidia card to an OCed one should yield margin of error differences and the same should be true for stock AMD card VS OCed one but, if comparisons between manufacturers are allot higher then margin of error, then you'll have your proof right there.

There's also this video that i found very interesting about CPU overhead in nVidia VS AMD:

 

[Kanan]

Later it was revealed that Adored's driver wasn't working correctly thus resulting in strange results for Nvidia gpus in dx12 with Ryzen. Still the video challenged reviewers to test Ryzen with AMD gpus aswell and not only latest Nvidia, that alone is a important step to get differentiated results.

 

[HTC]

Later it was revealed that Adored's driver wasn't working correctly thus resulting in strange results for Nvidia gpus in dx12 with Ryzen. Still the video challenged reviewers to test Ryzen with AMD gpus aswell and not only latest Nvidia, that alone is a important step to get differentiated results.

Really? Was not aware. Thanks for the heads up!

IMO, the CPU reviews should use 1 GPU from each camp and not from just one alone.

EDIT

Where did you find that? I'm not seeing it in the comments ???

 

[yogurt_21]

idk after 11 + years I'm pretty sure I can trust W1zzards reviews, After closer to 20 years of building, modding, and testing rigs, I also trust my own instincts.

They tell me the expectations for Ryzen were too high... WAYYYYY TO HIGH. While we all have great memories of AMD's Athlon 64 trashing Intel, that was pretty much the only time that happened. The Athlon XP was a great cpu, but it traded blows with the pentium 4 and sure performance gains per clock were higher, but you could clock the hell out of the P4's.

The Athlons themselves were decent, but did nothing to unseat the P3's. The K5/K6's were a good value, but the P2 trashed them. Amd had no answer for the pentium when it came out, back then they hadn't even fully designed a cpu in house yet. So they relied on leftovers and were still pushing 486 modeled cpus and wouldn't launch the K5 for another 2 years.

All the way through AMD had gradually improved against Intel. From not having any answer at all, to being thrashed, to competing per clock, to competing overall and winning per clock, to winning overall.

At no point did they come out of nowhere to beat Intel.

That was a stupid expectation.

I saw it in even the longest tenured forum members. Coming out and saying its trash if it doesn't beat Intel because...well they didn't really have any good reasons.

Value is determined at many levels, if it wasn't there wouldn't have been an AMD. Competing in certain areas and failing in others is pretty much the Athlon. It was a good cpu. I still have my 950MHZ in a box in my office. I loved that thing when I got it. For 100$ you couldn't beat it. And that was the key, a young college kid with a few dollars to his name could make himself a pretty awesome rig with AMD and ATI back in the day. 500$ out the door and you 1280x1024 gaming machine was playing all the latest titles.

Ryzen is the Athlon, a great step in the right direction. But you all forget that it wasn't until the Athlon XP series that AMD really started to put up the gaming numbers. And it wasn't until my 3500+ Newcastle that I really got to feel what it was like to have be cpu dominant.

that was fleeting. 2003-2006. Then Conroe and the ATI acquisition destroyed AMD.

I don't think there was anything wrong with how the cpus are tested. I think the expectations were wrong. If AMD continues this way maybe we'll just get another Athlon XP followed by Athlon 64 style rise from the ashes. Or not. Either way ryzen is a solid chip that just isn't the gaming king. Oh well.

 

[HTC]

idk after 11 + years I'm pretty sure I can trust W1zzards reviews, After closer to 20 years of building, modding, and testing rigs, I also trust my own instincts.

They tell me the expectations for Ryzen were too high... WAYYYYY TO HIGH. While we all have great memories of AMD's Athlon 64 trashing Intel, that was pretty much the only time that happened. The Athlon XP was a great cpu, but it traded blows with the pentium 4 and sure performance gains per clock were higher, but you could clock the hell out of the P4's.

The Athlons themselves were decent, but did nothing to unseat the P3's. The K5/K6's were a good value, but the P2 trashed them. Amd had no answer for the pentium when it came out, back then they hadn't even fully designed a cpu in house yet. So they relied on leftovers and were still pushing 486 modeled cpus and wouldn't launch the K5 for another 2 years.

All the way through AMD had gradually improved against Intel. From not having any answer at all, to being thrashed, to competing per clock, to competing overall and winning per clock, to winning overall.

At no point did they come out of nowhere to beat Intel.

That was a stupid expectation.

I saw it in even the longest tenured forum members. Coming out and saying its trash if it doesn't beat Intel because...well they didn't really have any good reasons.

Value is determined at many levels, if it wasn't there wouldn't have been an AMD. Competing in certain areas and failing in others is pretty much the Athlon. It was a good cpu. I still have my 950MHZ in a box in my office. I loved that thing when I got it. For 100$ you couldn't beat it. And that was the key, a young college kid with a few dollars to his name could make himself a pretty awesome rig with AMD and ATI back in the day. 500$ out the door and you 1280x1024 gaming machine was playing all the latest titles.

Ryzen is the Athlon, a great step in the right direction. But you all forget that it wasn't until the Athlon XP series that AMD really started to put up the gaming numbers. And it wasn't until my 3500+ Newcastle that I really got to feel what it was like to have be cpu dominant.

that was fleeting. 2003-2006. Then Conroe and the ATI acquisition destroyed AMD.

I don't think there was anything wrong with how the cpus are tested. I think the expectations were wrong. If AMD continues this way maybe we'll just get another Athlon XP followed by Athlon 64 style rise from the ashes. Or not. Either way ryzen is a solid chip that just isn't the gaming king. Oh well.

While i mostly agree with what you said, it seems you failed to notice that this thread is about the methodology of CPU benchmarking, regardless of the CPU used.

Sure: most videos here are of Ryzen VS Intel but that's because it's those that have been showing inconsistencies previously not found in others.

 

[EarthDog]

Weird... is this about CPUs or GPUs?

 

[yogurt_21]

While i mostly agree with what you said, it seems you failed to notice that this thread is about the methodology of CPU benchmarking, regardless of the CPU used.

Sure: most videos here are of Ryzen VS Intel but that's because it's those that have been showing inconsistencies previously not found in others.

you cannot talk about the methodology without talking about the products, their histories, and the current debates that make the misconceptions you're trying to point out.. Ryzen is a heavy part of this thread. deal with it.

 

[HTC]

Weird... is this about CPUs or GPUs?

CPUs, but GPUs used have been affecting the CPUs scores, when they should not.

 

[Kanan]

Really? Was not aware. Thanks for the heads up!

IMO, the CPU reviews should use 1 GPU from each camp and not from just one alone.

EDIT

Where did you find that? I'm not seeing it in the comments ???

You're welcome. Watch this video (direct reaction to Adored's, they talk to eachother as well):

 

[HTC]

You're welcome. Watch this video (direct reaction to Adored's, they talk to eachother as well):

Had seen the video but not the comments: thanks for the heads up.