Tuesday, October 9th 2018

Intel's 9th Gen Core Gaming Benchmarks Flawed and Misleading

At its 9th Generation Core processor launch extravaganza earlier this week, Intel posted benchmark numbers to show just how superior its processors are to AMD 2nd generation Ryzen "Pinnacle Ridge." PC enthusiasts worth their salt were quick to point out that Intel's numbers are both flawed and misleading as they misrepresent both test setups - by optimizing Intel processors beyond their out-of-the-box performance, and by running AMD processors with sub-optimal settings.

Intel paid Principled Technologies, a third-party performance testing agency, to obtain performance numbers comparing the Core i9-9900K with the Ryzen 7 2700X across a spectrum of gaming benchmarks, instead of testing the two chips internally, and posting their test setup data in end-notes, as if to add a layer of credibility/deniability to their charade. The agency posted its numbers that were almost simultaneously re-posted PCGamesN, gleaming the headline "Up to 50% Faster than Ryzen at Gaming." You could fertilize the Sahara with this data.
Right off the bat, we see Principled Technologies use a sub-optimal memory configuration for the Ryzen 7 2700X machine, saddling it with a dual-rank memory with all four memory slots populated, and running at stock memory speeds with the motherboard BIOS determining "stable" memory timings. AMD processors compensate for dual-rank / 4-module setups by either restricting memory clocks or loosening up memory timings in the interest of stability. Principled Technologies incompetently set the Ryzen setup's memory clocks to 2933 MHz, leaving the motherboard BIOS to find extremely loose memory timings to stabilize the memory clock.

In stark contrast to this, for the Core i9-9900K machine, the testers simply flicked the XMP profile of the Corsair Vengeance RGB DDR4-3000 memory kit, which ended up running at not just higher clocks, but also tighter timings (which have been tested by Corsair on an Intel platform to obtain the XMP certificate). They reinforced the memory by adjusting the frequency manually. This gives the Intel platform a significant performance advantage against AMD. Ryzen processors are more memory-sensitive than Intel, as DRAM clocks are synchronized with other clock domains such as the InfinityFabric clock, which determines the data-rate of communication between the two Zen Compute Complex (CCX) components on the 8-core "Pinnacle Ridge" die.

As if that wasn't bad enough, the white paper reveals that some of the games were tested on the Ryzen machine with the "game mode" enabled via Ryzen Master. What this does is localise a game to just one of the two CCX units, essentially turning the 8-core chip to quad-core. The game mode is known to have a negative performance impact on games that can use more than 4 cores, or which are memory bandwidth intensive. This is truly below the belt from Intel.

The next part of its deception was testing both setups at 1080p on "Ashes of the Singularity" CPU benchmark with medium settings, to obtain extremely suspicious performance numbers. When HardwareUnboxed used similar settings to compare their Core i7-8700K with the Ryzen 7 2700X (using sane memory settings for both setups), the performance numbers obtained were very different, and don't bode well for the credibility of their i9-9900K numbers. Without the unfair advantage to the i9-9900K, the Ryzen 7 2700X yields up to 18% higher frame-rates than what Intel's numbers suggest. The story repeats (albeit to a smaller degree), with most other benchmarks posted by Intel. "Assassin's Creed Origins" is another benchmark where Principled Technologies numbers paint the Intel 8700K at 36% faster than the 2700X, while in reality, the 8700K is more like 8% faster.

Normally, performance numbers released by hardware manufacturers at launch are disregarded by consumers as hardware launches are almost always simultaneously followed by independent reviewers being allowed to post their benchmark numbers. Off late, however, there is a worrying trend of hardware manufacturers launching their products with reviewer NDAs expiring weeks later, letting them solicit pre-orders on the basis of questionable performance data. In this case, Intel's gilded numbers release almost 2 weeks before the review NDA, and the Core i9-9900K is up for pre-order, in some places even at $540.

We strongly recommend you to wait until you read performance reviews from multiple tech publications before basing your purchase decisions. It's a foregone conclusion that the i9-9900K will be faster than the 2700X, as the i7-8700K already trades blows with it despite having two fewer cores. However, the percentage-difference in performance, and the cost-performance numbers put out by Intel for the upcoming chip, are extremely questionable at this point.

Update 19:55 UTC:
Intel provided following statement to GamersNexus regarding the testing
We are deeply appreciative of the work of the reviewer community and expect that over the coming weeks additional testing will continue to show that the 9th Gen Intel Core i9-9900K is the world's best gaming processor. PT conducted this initial testing using systems running in spec, configured to show CPU performance and has published the configurations used. The data is consistent with what we have seen in our labs, and we look forward to seeing the results from additional third party testing in the coming weeks.
Sources: Intel Benchmark Results & Methodology, HardwareUnboxed (YouTube)
Add your own comment

76 Comments on Intel's 9th Gen Core Gaming Benchmarks Flawed and Misleading

#26
ShurikN
iOOr here directly to the PDF
I forgot to mention this in my first post. The reason why TPU didn't post pictures of those shady results is because that would probably breach the NDA that they signed (otherwise they wouldn't care). Steve didn't sign an NDA (bought the review sample) therefore he can post whatever he can
Posted on Reply
#27
laszlo
they get paid and they delivered the stuff as requested...

question is how many other reviewers do the same....

unfortunately uneducated buyer will buy everything no matter if is worthy, needed or not, nothing new under the sun
Posted on Reply
#28
JalleR
ShurikNI forgot to mention this in my first post. The reason why TPU didn't post pictures of those shady results is because that would probably breach the NDA that they signed (otherwise they wouldn't care). Steve didn't sign an NDA (bought the review sample) therefore he can post whatever he can
Oh... Yes that makes sense. :)
Posted on Reply
#29
Dave65
About what id expect out of Intel!
Intel, you're dead to me:shadedshu:
Posted on Reply
#30
R0H1T
So Intel's worse than the cheating scums we all thought knew they are :laugh:
Posted on Reply
#31
Robcostyle
kingsWell, If people pre-order something, it's entirely their fault. People should never pre-order something without first knowing all the facts.
There's thousands of them - I'm taling about people preordering stuff, willing to pay ahead, or pay for whatever made "wow" effect on'em, even without knowing what that thing is for. Hundreds of pumpkinheads braged about preordering 2080\(Ti).
And that makes their "faults" being our problems - for instance, the price rise we see today. Market is oriented to the majority - and the majority isn't used to use their head before paying a bill.
And that's sad.
Posted on Reply
#32
bug
R0H1TSo Intel's worse than the cheating scums we all thought knew they are :laugh:
Only if you enjoy jumping to conclusions.
Sure, the setup was suboptimal. But at this point we don't know how much it actually influenced the results. For all we know, the results may not change much if you tighten the timings for AMD, because you'd be running the memory at lower speeds and IF doesn't really like that anyway. And it's not like running with 4 memory banks populated is an invalid scenario in the real world.
Posted on Reply
#33
mcraygsx
Let us not forget we are comparing a $275 CPU against a CPU that cost more then $550.99. These results are not very far off from each other anyways and consumers will likely run into GPU bottleneck at higher resolutions. Not everyone spends thousands of dollars to play AAA titles at 1080p
Posted on Reply
#34
medi01
jboydgolferi hope AMD builds a better chip.
AMD's CPUs perform on par with Intels, sometimes 10% behind (normally single threaded stuff or stuff that depends on nvidia's Intel optimizations in drivers) sometimes quite ahead, especially in multithreaded apps, even with core for core . AMD's power consumption is on par or better, despite inferior fab process.


I hope people get real about AMD/Intel/NGreedia oferrings.
bugSure, the setup was suboptimal. But at this point we don't know how much it actually influenced the results.
Ah. Because... Erm... Because if "not much" it's only 'not much cheating" I guess.
Logical.
Posted on Reply
#35
bug
medi01Ah. Because... Erm... Because if "not much" it's only 'not much cheating" I guess.
Logical.
No. It's because even with optimal settings, the outcome may still be the same.
But when it comes to faulting Intel/Nvidia, we don't need numbers any longer, just an accusation will do. Nevermind that we're already discarding reviewer negligence/incompetence and are already assuming Intel asked them to set the benchmark up as they did.
Posted on Reply
#36
R0H1T
bugNo. It's because even with optimal settings, the outcome may still be the same.
But when it comes to faulting Intel/Nvidia, we don't need numbers any longer, just an accusation will do. Nevermind that we're already discarding reviewer negligence/incompetence and are already assuming Intel asked them to set the benchmark up as they did.
What about the accusation that some(?) of the tests were run with 2700x as a quad core, did hardware unboxed make that up?

Now that you mention it, lemme see ~ after spectre/meltdown, shouting ~ hey they were affected as well. Except meltdown is not a Zen flaw.

Amdflaws.com ~ did anyone get to the bottom of that?

Faking 28c 5GHz record breaking demo, which was then rebadged as a Xeon (on socket P) & will be released shortly, not as HEDT though!

Nope, Intel's always guilty in my eyes. I don't like them & I don't like giving them free passes every time they do "whoops, we did it again" :)
Posted on Reply
#37
ironwolf
Update from Hardware Unboxed...
The benchmarks carried out by Principled Technologies are even more bogus than we first thought. A few viewers pointed out that the Ryzen 7 2700X was listed as tested in the “Game Mode” within the Ryzen Master software and I foolishly thought they might have just made a simple copy and paste error in their document as they would have used this mode for the 2950X. This does explain why the Threadripper CPUs were faster than the 2700X in every test.

What this means is a CCX module in the 2700X was completely disabled, essentially turning it into a quad-core. I’ve gone ahead and re-run the XMP 2933 test with Game Mode enabled and now I’m getting results that are within the margin of error to those published by Principled Technologies.
Rest of the update: www.patreon.com/posts/21950120
Posted on Reply
#39
Vayra86
Durvelle27Why does AMD need to build a better chip

It has be proven with even previous gen that intel on average is barely 10% faster than AMDs offering while having much higher clocks

Are people that naive to rave about a measly performance gain but a bigger hand in your pockets
10%? Source pls. I csn show you sources where Intel extracts FPS precisely relative to its clockspeed advantage and its no secret either...

There is a large gap but you only notice it with high refresh rates. For 60 fps most CPUs are fine.

That proof you speak of only exists in a performance summary. When you look at scenarios where there is no restricting factor such as a GPU, the single thread gap is still quite big and its almost entirely due to clocks.

Regardless, Intel should be burned for this and they are. Good riddance
Posted on Reply
#40
londiste
Game Mode is the dick move.

Memory speeds are the usual manufacturer's benchmark thing where they find the weakest link in configuration that is technically legit. Ryzen spec for memory speed support says 2933 only for two modules. For 4 modules, the supported speed drops to 2133 and even 1866 if the modules are dual-rank.
Posted on Reply
#41
Vayra86
mcraygsxLet us not forget we are comparing a $275 CPU against a CPU that cost more then $550.99. These results are not very far off from each other anyways and consumers will likely run into GPU bottleneck at higher resolutions. Not everyone spends thousands of dollars to play AAA titles at 1080p
Not everyone but still a vast majority really does especially with GPU priced through the roof. I reckon its at least as much as people on 4K, likely more.
Posted on Reply
#42
bug
R0H1TNope, Intel's always guilty in my eyes. I don't like them & I don't like giving them free passes every time they do "whoops, we did it again" :)
That is your right and I won't dispute that.
But calling them "scum" just because you choose to assume the worst about them is something else.
Posted on Reply
#43
Salty_sandwich
I don't care who's got fastest CPU my system spec says it all lol
just as long as this little PC does what I need it to do, gone are the days I want the best or even need it lol
fun to read all the banter people have to go through just to feel the best :)
Posted on Reply
#44
Xaled
Intel for long time were the dirtiest tech company on earth, before nVidia take the the crown. Just remember the recent fake israeli cyber security company thing.. nVidia now is the devil and they are about to kill the whole pc gaming industry..
Posted on Reply
#45
R0H1T
bugThat is your right and I won't dispute that.
But calling them "scum" just because you choose to assume the worst about them is something else.
That's because even at their (innovative) best, core 2 era, they've generally been anti consumer & certainly anti competition. There's documented history about their misdeeds, which I'm sure you'd be well aware of. So if not scum ~ sure you can call them whatever you'd like to, but the essence remains same AFAIK & semantics, in this case, don't matter.
Posted on Reply
#46
bug
R0H1TThat's because even at their (innovative) best, core 2 era, they've generally been anti consumer & certainly anti competition. There's documented history about their misdeeds, which I'm sure you'd be well aware of. So if not scum ~ sure you can call them whatever you'd like to, but the essence remains same AFAIK & semantics, in this case, don't matter.
They matter in the context of a civil conversation.
Posted on Reply
#47
iO
The main thing you can take away from their "tests" is that the 9900K only achieved 5.5% higher FPS on average than the 8700K.
Posted on Reply
#48
Supercrit
Didn't Intel uses a Noctua NH14S cooler while AMD had to content with stock? Not that stock AMD cooler is bad, but compare the free cooler with a premium one is a dick move.
Posted on Reply
#49
londiste
SupercritDidn't Intel uses a Noctua NH14S cooler while AMD had to content with stock? Not that stock AMD cooler is bad, but compare the free cooler with a premium one is a dick move.
They did not compare temperatures or noise. 2700X at stock doesn't throttle under Prism.

Why so much noise about a stupid manufacturer "test results"? These have always been marketing and little else.
Posted on Reply
#50
Patriot
bugIt's not like AMD didn't "leak" those video processing benchmarks over and over before the Zen release, making it look like it will destroy Intel in IPC.
As I have said before, absolutely no reason to be shocked. Apply the mandatory grain of salt or disregard these "leaks" entirely. Goes a long way preserving one's sanity ;)
AMD presented best use case for their chips, but they didnt fuck over the intel setup it compared against. The results were real and repeatable. They have continued to push a mixed use case for their chips.
Posted on Reply
Add your own comment
Dec 19th, 2024 06:13 EST change timezone

New Forum Posts

Popular Reviews

Controversial News Posts