Tuesday, October 9th 2018
Intel's 9th Gen Core Gaming Benchmarks Flawed and Misleading
At its 9th Generation Core processor launch extravaganza earlier this week, Intel posted benchmark numbers to show just how superior its processors are to AMD 2nd generation Ryzen "Pinnacle Ridge." PC enthusiasts worth their salt were quick to point out that Intel's numbers are both flawed and misleading as they misrepresent both test setups - by optimizing Intel processors beyond their out-of-the-box performance, and by running AMD processors with sub-optimal settings.
Intel paid Principled Technologies, a third-party performance testing agency, to obtain performance numbers comparing the Core i9-9900K with the Ryzen 7 2700X across a spectrum of gaming benchmarks, instead of testing the two chips internally, and posting their test setup data in end-notes, as if to add a layer of credibility/deniability to their charade. The agency posted its numbers that were almost simultaneously re-posted PCGamesN, gleaming the headline "Up to 50% Faster than Ryzen at Gaming." You could fertilize the Sahara with this data.
Right off the bat, we see Principled Technologies use a sub-optimal memory configuration for the Ryzen 7 2700X machine, saddling it with a dual-rank memory with all four memory slots populated, and running at stock memory speeds with the motherboard BIOS determining "stable" memory timings. AMD processors compensate for dual-rank / 4-module setups by either restricting memory clocks or loosening up memory timings in the interest of stability. Principled Technologies incompetently set the Ryzen setup's memory clocks to 2933 MHz, leaving the motherboard BIOS to find extremely loose memory timings to stabilize the memory clock.
In stark contrast to this, for the Core i9-9900K machine, the testers simply flicked the XMP profile of the Corsair Vengeance RGB DDR4-3000 memory kit, which ended up running at not just higher clocks, but also tighter timings (which have been tested by Corsair on an Intel platform to obtain the XMP certificate). They reinforced the memory by adjusting the frequency manually. This gives the Intel platform a significant performance advantage against AMD. Ryzen processors are more memory-sensitive than Intel, as DRAM clocks are synchronized with other clock domains such as the InfinityFabric clock, which determines the data-rate of communication between the two Zen Compute Complex (CCX) components on the 8-core "Pinnacle Ridge" die.
As if that wasn't bad enough, the white paper reveals that some of the games were tested on the Ryzen machine with the "game mode" enabled via Ryzen Master. What this does is localise a game to just one of the two CCX units, essentially turning the 8-core chip to quad-core. The game mode is known to have a negative performance impact on games that can use more than 4 cores, or which are memory bandwidth intensive. This is truly below the belt from Intel.
The next part of its deception was testing both setups at 1080p on "Ashes of the Singularity" CPU benchmark with medium settings, to obtain extremely suspicious performance numbers. When HardwareUnboxed used similar settings to compare their Core i7-8700K with the Ryzen 7 2700X (using sane memory settings for both setups), the performance numbers obtained were very different, and don't bode well for the credibility of their i9-9900K numbers. Without the unfair advantage to the i9-9900K, the Ryzen 7 2700X yields up to 18% higher frame-rates than what Intel's numbers suggest. The story repeats (albeit to a smaller degree), with most other benchmarks posted by Intel. "Assassin's Creed Origins" is another benchmark where Principled Technologies numbers paint the Intel 8700K at 36% faster than the 2700X, while in reality, the 8700K is more like 8% faster.
Normally, performance numbers released by hardware manufacturers at launch are disregarded by consumers as hardware launches are almost always simultaneously followed by independent reviewers being allowed to post their benchmark numbers. Off late, however, there is a worrying trend of hardware manufacturers launching their products with reviewer NDAs expiring weeks later, letting them solicit pre-orders on the basis of questionable performance data. In this case, Intel's gilded numbers release almost 2 weeks before the review NDA, and the Core i9-9900K is up for pre-order, in some places even at $540.
We strongly recommend you to wait until you read performance reviews from multiple tech publications before basing your purchase decisions. It's a foregone conclusion that the i9-9900K will be faster than the 2700X, as the i7-8700K already trades blows with it despite having two fewer cores. However, the percentage-difference in performance, and the cost-performance numbers put out by Intel for the upcoming chip, are extremely questionable at this point.
Update 19:55 UTC:
Intel provided following statement to GamersNexus regarding the testing
Sources:
Intel Benchmark Results & Methodology, HardwareUnboxed (YouTube)
Intel paid Principled Technologies, a third-party performance testing agency, to obtain performance numbers comparing the Core i9-9900K with the Ryzen 7 2700X across a spectrum of gaming benchmarks, instead of testing the two chips internally, and posting their test setup data in end-notes, as if to add a layer of credibility/deniability to their charade. The agency posted its numbers that were almost simultaneously re-posted PCGamesN, gleaming the headline "Up to 50% Faster than Ryzen at Gaming." You could fertilize the Sahara with this data.
In stark contrast to this, for the Core i9-9900K machine, the testers simply flicked the XMP profile of the Corsair Vengeance RGB DDR4-3000 memory kit, which ended up running at not just higher clocks, but also tighter timings (which have been tested by Corsair on an Intel platform to obtain the XMP certificate). They reinforced the memory by adjusting the frequency manually. This gives the Intel platform a significant performance advantage against AMD. Ryzen processors are more memory-sensitive than Intel, as DRAM clocks are synchronized with other clock domains such as the InfinityFabric clock, which determines the data-rate of communication between the two Zen Compute Complex (CCX) components on the 8-core "Pinnacle Ridge" die.
As if that wasn't bad enough, the white paper reveals that some of the games were tested on the Ryzen machine with the "game mode" enabled via Ryzen Master. What this does is localise a game to just one of the two CCX units, essentially turning the 8-core chip to quad-core. The game mode is known to have a negative performance impact on games that can use more than 4 cores, or which are memory bandwidth intensive. This is truly below the belt from Intel.
The next part of its deception was testing both setups at 1080p on "Ashes of the Singularity" CPU benchmark with medium settings, to obtain extremely suspicious performance numbers. When HardwareUnboxed used similar settings to compare their Core i7-8700K with the Ryzen 7 2700X (using sane memory settings for both setups), the performance numbers obtained were very different, and don't bode well for the credibility of their i9-9900K numbers. Without the unfair advantage to the i9-9900K, the Ryzen 7 2700X yields up to 18% higher frame-rates than what Intel's numbers suggest. The story repeats (albeit to a smaller degree), with most other benchmarks posted by Intel. "Assassin's Creed Origins" is another benchmark where Principled Technologies numbers paint the Intel 8700K at 36% faster than the 2700X, while in reality, the 8700K is more like 8% faster.
Normally, performance numbers released by hardware manufacturers at launch are disregarded by consumers as hardware launches are almost always simultaneously followed by independent reviewers being allowed to post their benchmark numbers. Off late, however, there is a worrying trend of hardware manufacturers launching their products with reviewer NDAs expiring weeks later, letting them solicit pre-orders on the basis of questionable performance data. In this case, Intel's gilded numbers release almost 2 weeks before the review NDA, and the Core i9-9900K is up for pre-order, in some places even at $540.
We strongly recommend you to wait until you read performance reviews from multiple tech publications before basing your purchase decisions. It's a foregone conclusion that the i9-9900K will be faster than the 2700X, as the i7-8700K already trades blows with it despite having two fewer cores. However, the percentage-difference in performance, and the cost-performance numbers put out by Intel for the upcoming chip, are extremely questionable at this point.
Update 19:55 UTC:
Intel provided following statement to GamersNexus regarding the testing
We are deeply appreciative of the work of the reviewer community and expect that over the coming weeks additional testing will continue to show that the 9th Gen Intel Core i9-9900K is the world's best gaming processor. PT conducted this initial testing using systems running in spec, configured to show CPU performance and has published the configurations used. The data is consistent with what we have seen in our labs, and we look forward to seeing the results from additional third party testing in the coming weeks.
76 Comments on Intel's 9th Gen Core Gaming Benchmarks Flawed and Misleading
This has to be the best thing ever.
A more interesting result/test is the 8700K versus 9700K, and getting some data on the average achieved ambient overclocks on each one. I'm really curious if the 9700K will clock higher in some sort of reliable way, it being a soldered CPU and non-HT @ 8c. Even though its not apples/apples, it will be nice to see how much the 'toothpaste' *truly* mattered for all these years and infinite number of whine topics about it. Honestly I think its going to be 100-200mhz at best and it might not even be a consistent/reliable improvement at that.
Despite Intel's dark behaviour to review its own product, this news just makes me run even farther from AMDs platform.
Either way more people buy Intel because they believe this marketing BS (And yep, Intel is desperate against Ryzen so they have to market their tiny gaming edge as MASSIVE) and they buy Intel because it says Intel on it.
AMD Ryzen cannot win, because it is up against stupidity.*
*Disclaimer: for Super High Refresh rate gaming (144Hz+) and especially in competitive games the advantage intel has in raw FPS in some titles might be worth it for some people. I put this here because some folks are going to go mad at me for even suggesting Intel products aren't worth it.
I game on a R5 2600 at 4.1 with some pretty mediocre timings on my RAM (its a heavy OC though) and I'm very rarely seeing below 62 fps (my cap) in video games. And at the moment I am playing Fallout 4, Warframe and Far Cry 5. But I have played a lot of games. Sometimes I get dips below 60 in FC5 but it's few and far between and from what i heard Intel CPUs also get these dips.
Why does it bother me so much? It is because by nature I am a passionate person. I don't care just about AMD as a company, I care about a free and open market with good consumer choice and fair prices for all people. Blindly supporting one company because of brand name is damaging that. And the tech market is something I care about as it is one of my hobbies.
Sorry for the long post.
PS
But then I realized, it's not AMD that they necessarily have to discredit in order to win customers, it's their own products that they target. They have to convince somehow 8700K and 7700K users that is time to upgrade.
Intel is being hilarious and pathetic as always.
*fanboy rears his head by making one party into several ;)
I did this on both of my Ryzen systems and presto, magic! RAM speed and timings matching the XMP profile....
Amazingly these steps also had to be followed on my Intel system as well..
The result... Identical memory speeds / timings on both systems when the same RAM is used.
Anyway... it's just one benchmark. Whoever buys based on one benchmark, gets just what they deserve.
Somehow Intel sends them 9900K just for fun? Just for playing around? Even though they get 9900K before anyone else? Wow... Good guy Intel, indeed. /s