Wednesday, October 10th 2018
Principled Technologies Comments on their Intel Processor Study
Today, we have seen several reports that suggested Principled Technologies (PT) published misleading information in our recent study comparing Intel's gaming processors to AMD's. We apologize for our delay in responding, but it's been a busy day, and we wanted to be as thorough as possible in addressing inquiries concerning our testing. We'll address specific questions and share more detail on our methodology in a moment, but we first must respond directly to attempts to call our integrity into question.
For almost 16 years, we have tested products for our clients because they trust our integrity. We have worked not just for any one company but for dozens of the leading technology firms, including rivals such as Intel and AMD, Microsoft and Google, Dell and HP, and many others.
Those clients trust PT in part because our integrity and our technical knowledge are beyond reproach. We work hard to be the best in both of those areas. We chose our company name to emphasize our commitments to both technology and our principles. (And, accusers saying we are only in this for the money obviously haven't read our book, Limit Your Greed!)
Before going further, we thus must categorically deny any dishonesty in our work on this project for Intel or in any of our other projects.
Now that we've gotten that off our chests, let's address the specific questions from recent videos as well as subsequent posts and tweets.
Project overview
An overview of the project will provide useful context.
Our overall goal - and Intel's specific request for this project - was to create as level a playing field as possible for comparing the AMD and Intel processors as the majority of the gaming market would likely use them. To do that, we built and configured 16 systems for this comparative testing; we had two of each processor/motherboard configuration. We matched all components where possible, the only variances being the motherboards, CPUs, and CPU coolers. (Full details are in our interim report.)
In an effort to be very transparent, we published our interim summary report on Oct 8, 2018. We will continue to be transparent and responsive to any questions.
Responses to inquiries
We have received a number of inquiries regarding the testing methodology we used and the potential for bias in favor of Intel. We are providing additional information to be as transparent as possible and to help allay these concerns.
The following list summarizes many of the inquiries we have received and our responses. (We are continuing to work on addressing additional inquiries.)
MSI Z390-A Pro motherboard (i9-9900K)
Asus Prime X399-A (Threadripper 2990WX, Threadripper 2950X)
Quality settings: We configured all games to use the "High" or equivalent preset, versus "Ultra" or other presets, also to emphasize CPU over GPU performance. In the case where there were only three presets, we chose the top preset.
Clarification of various installation questions: We installed all games using Steam or the Microsoft Store, and fully updated with the latest patches.
Motherboards: Re a Twitter comment from Cyber Cat @0xCats, "Hey @AMD Apparently according to @PrincipledTech @Intel is able to run Ryzen & Threadripper Chips on Z370 and Z390": Thanks for catching that copy/paste error in our configuration info. We made an error there. The correct processor/motherboard/BIOS version specs for the AMD procs we tested are the following: Ryzen 7 2700X/ASUS PRIME X470-PRO/4024 and Threadripper 2950X & 2990WX/ASUS PRIME X399-A/0807. We apologize for the error and will post a revised version (with changes noted) soon.
Because our goal is always to do the right thing and get the answers right, we are currently doing additional testing. We will share that data and will certainly call out if something is significantly different from what we've already published.
We are confident in our test methodology and results. We welcome questions and we are doing our best to respond to questions from our interim report, but doing so takes time. We will add responses if other issues come up.
Thanks for listening.
For almost 16 years, we have tested products for our clients because they trust our integrity. We have worked not just for any one company but for dozens of the leading technology firms, including rivals such as Intel and AMD, Microsoft and Google, Dell and HP, and many others.
Those clients trust PT in part because our integrity and our technical knowledge are beyond reproach. We work hard to be the best in both of those areas. We chose our company name to emphasize our commitments to both technology and our principles. (And, accusers saying we are only in this for the money obviously haven't read our book, Limit Your Greed!)
Before going further, we thus must categorically deny any dishonesty in our work on this project for Intel or in any of our other projects.
Now that we've gotten that off our chests, let's address the specific questions from recent videos as well as subsequent posts and tweets.
Project overview
An overview of the project will provide useful context.
Our overall goal - and Intel's specific request for this project - was to create as level a playing field as possible for comparing the AMD and Intel processors as the majority of the gaming market would likely use them. To do that, we built and configured 16 systems for this comparative testing; we had two of each processor/motherboard configuration. We matched all components where possible, the only variances being the motherboards, CPUs, and CPU coolers. (Full details are in our interim report.)
In an effort to be very transparent, we published our interim summary report on Oct 8, 2018. We will continue to be transparent and responsive to any questions.
Responses to inquiries
We have received a number of inquiries regarding the testing methodology we used and the potential for bias in favor of Intel. We are providing additional information to be as transparent as possible and to help allay these concerns.
The following list summarizes many of the inquiries we have received and our responses. (We are continuing to work on addressing additional inquiries.)
- Use of "Game Mode" on the AMD Ryzen 7 2700X: Some inquiries we have received concern the use of the Ryzen utility and the number of active cores in the AMD-based systems. Based on AMD's recommendations and our initial testing on the Threadripper processors, we found installing the AMD Ryzen Master utility and enabling the Game Mode increased most results. For consistency purposes, we did that for all AMD systems across Threadripper and Ryzen. We are now doing additional testing with the AMD systems in Creator Mode. We will update the report with the new results.
- Cooler choice: We chose Noctua for the CPU coolers, due to having almost identical systems in the NH-U14S (Intel) and NH-U14S TR4-SP3 (AMD), which allowed us to maintain a comparable thermal profile. Because we were not performing any overclocking on any configuration, and because AMD has said it was a good cooler, we stuck with the stock AMD Ryzen 7 2700X Wraith Prism cooler.
- Memory speeds: To have complete parity across all systems, and to allow the Intel Core i9 X-series and AMD Ryzen Threadripper to fully utilize memory bandwidth, we used four 16 GB DDR4 DIMMs on all configurations. We took the following memory configuration steps:
MSI Z390-A Pro motherboard (i9-9900K)
- Load Optimized BIOS defaults
- Enabled: Extreme Memory Profile (X.M.P.)
- DRAM Frequency set to DDR4-2666
- Asus Prime X299-Deluxe motherboard (i9-9900X ,i9-9980XE)
- Load Optimized BIOS defaults
- Enabled: Extreme Memory Profile (X.M.P.)
- Disabled ASUS MultiCore Enhancement to use stock Intel multicore settings
- DRAM Frequency set to DDR4-2666
- Installed Intel Turbo Boost Maxdriver/utility
- Load Optimized BIOS defaults
- Enabled: Extreme Memory Profile(X.M.P.)
- Disabled ASUS MultiCore Enhancement to use stock Intel multicore settings
- DRAM Frequency set to DDR4-2666
- Power saving & Performance mode, set to Performance
Asus Prime X399-A (Threadripper 2990WX, Threadripper 2950X)
- Load Optimized BIOS defaults
- Verify that D.O.C.P is selected for AMD-equivalent memory settings to XMP
- Performance Enhancer, set to Default
- Disabled overclocking enhancement
- DRAM frequency set to DDR4-2933
- Set Core Performance Boost to Auto
- Set performance bias to None
- Installed Ryzen Master utility
- Load Optimized BIOS defaults
- Verify that D.O.C.P is selected for AMD-equivalent memory settings to XMP
- DRAM frequency set to DDR4-2933
- Set performance bias to None
- Installed Ryzen Master utility
Quality settings: We configured all games to use the "High" or equivalent preset, versus "Ultra" or other presets, also to emphasize CPU over GPU performance. In the case where there were only three presets, we chose the top preset.
Clarification of various installation questions: We installed all games using Steam or the Microsoft Store, and fully updated with the latest patches.
Motherboards: Re a Twitter comment from Cyber Cat @0xCats, "Hey @AMD Apparently according to @PrincipledTech @Intel is able to run Ryzen & Threadripper Chips on Z370 and Z390": Thanks for catching that copy/paste error in our configuration info. We made an error there. The correct processor/motherboard/BIOS version specs for the AMD procs we tested are the following: Ryzen 7 2700X/ASUS PRIME X470-PRO/4024 and Threadripper 2950X & 2990WX/ASUS PRIME X399-A/0807. We apologize for the error and will post a revised version (with changes noted) soon.
Because our goal is always to do the right thing and get the answers right, we are currently doing additional testing. We will share that data and will certainly call out if something is significantly different from what we've already published.
We are confident in our test methodology and results. We welcome questions and we are doing our best to respond to questions from our interim report, but doing so takes time. We will add responses if other issues come up.
Thanks for listening.
41 Comments on Principled Technologies Comments on their Intel Processor Study
“ Principled Technologies Benchmark Disclosure: Intel is a sponsor and member of the BenchmarkXPRT* Development Community, and was the major developer of the XPRT* family of benchmarks. Principled Technologies is the publisher of the XPRT* family of benchmarks. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases”
As a technical writer, I can see why they did what they did, and actually I kind of agree with how they did things.
Now, would I have run tests like that? NOPE.
Will I run tests like that? NOPE.
Could I have done it differently, and would Intel still have won the majority of benchmarks?
You bet your ass I could.
And that... that's what makes it bad for me.
But do I care? NOPE...
...and because it was used for marketing. It just all the more reason for people like me to do what we do.
BTW, amateur youtubers are some of the worst sources of info there is, and the fact you mention them... man... they often have it so misconstrued, and are doing their thing to make money with views, just like Principled Technologies does. I see the youtubers as far worse than PT is. But they serve a purpose too. Even negative attention is attention, for marketing.
Like...Look... everyone's talking about Intel's latest, and there's no real reviews, and you can't buy them yet.
PT did an awesome job. :p
RE: PT & the i9 though, They’re up for pre-order, and there’s a single source of benchmarks for interested parties. That’s my entire point, amateur youtuber’s are the (legitimate) benchmark for all that is substandard and yet compared to this “study” they come out looking like perfectionists, if you conclude the problems with this report were incompetence.
If you conclude they knew what they were doing, then it’s difficult to excuse such amateur errors conveniently placed as they were. From Seasoned vet to rookie and back again repeatedly? No clue something might be up with a test showing 1500% > performance for the i9, by the same person competent enough to set up such a test and competent enough to know to look into all the little motherboard specific nuances necessary to unleash the Intel/AMD sku’s but they just happen to accidentally cripple the performance of the SINGLE-MOST relevant AMD part, the most direct competitor, the one cpu that you or I would have been most focused on ensuring was properly respresented before putting our name on it.
And then thoroughly documenting every step so that any half competent amateur youtuber’s can attempt to reproduce the results and will indeed see the same thing, if they follow them to a tee.
And Intel has the balls to come out and co-sign it and say they see the same in their labs without any further clarification, while critical pre-order hours pass by literally the only benchmark available on the internet conveniently shows an i9 that miraculously performs exponentially better than even intel’s Own skylake-x 8 core sku’s.
Ok yea I guess that’s reasonable, seems like it belongs in a Lemony Snicket tale to me though.
Great thoughts though; I'll have to digest these and comment later.NO, but really, the recent NVidia launch and pre-order time benefitted NVidia just as much as this might benefit Intel. Paper launches are dumb, but NVidia showed how great they can be.
Oh yeah, I guess Apple and many other companies do it too. Funny how we make circuits on glass and circuit boards interesting these days, huh?
4x16 probably because they had a bunch of these in hand. 4x8 (or 4x4) would not have made much of a difference in context of test results.
On the other hand I suspect press release is written by someone (or with input from someone) who does.
I mean, I get it. Ryzen really does not like 4 memory modules. But it does not invalidate the idea of using the same 4 modules across all different systems. If they would use 2 modules for some systems and 4 for others there would be different questions about specific modules, memory capacity etc.
Incredible clumsiness. Even having to correct actual errors in their writing in the same piece where they say they are the greatest. LMAO
It is common knowledge that to call AMD's press/marketing strategy amature, is actually a big compliment. AMD have always been their second worst enemy.