Wednesday, October 10th 2018
Principled Technologies Comments on their Intel Processor Study
Today, we have seen several reports that suggested Principled Technologies (PT) published misleading information in our recent study comparing Intel's gaming processors to AMD's. We apologize for our delay in responding, but it's been a busy day, and we wanted to be as thorough as possible in addressing inquiries concerning our testing. We'll address specific questions and share more detail on our methodology in a moment, but we first must respond directly to attempts to call our integrity into question.
For almost 16 years, we have tested products for our clients because they trust our integrity. We have worked not just for any one company but for dozens of the leading technology firms, including rivals such as Intel and AMD, Microsoft and Google, Dell and HP, and many others.
Those clients trust PT in part because our integrity and our technical knowledge are beyond reproach. We work hard to be the best in both of those areas. We chose our company name to emphasize our commitments to both technology and our principles. (And, accusers saying we are only in this for the money obviously haven't read our book, Limit Your Greed!)
Before going further, we thus must categorically deny any dishonesty in our work on this project for Intel or in any of our other projects.
Now that we've gotten that off our chests, let's address the specific questions from recent videos as well as subsequent posts and tweets.
Project overview
An overview of the project will provide useful context.
Our overall goal - and Intel's specific request for this project - was to create as level a playing field as possible for comparing the AMD and Intel processors as the majority of the gaming market would likely use them. To do that, we built and configured 16 systems for this comparative testing; we had two of each processor/motherboard configuration. We matched all components where possible, the only variances being the motherboards, CPUs, and CPU coolers. (Full details are in our interim report.)
In an effort to be very transparent, we published our interim summary report on Oct 8, 2018. We will continue to be transparent and responsive to any questions.
Responses to inquiries
We have received a number of inquiries regarding the testing methodology we used and the potential for bias in favor of Intel. We are providing additional information to be as transparent as possible and to help allay these concerns.
The following list summarizes many of the inquiries we have received and our responses. (We are continuing to work on addressing additional inquiries.)
MSI Z390-A Pro motherboard (i9-9900K)
Asus Prime X399-A (Threadripper 2990WX, Threadripper 2950X)
Quality settings: We configured all games to use the "High" or equivalent preset, versus "Ultra" or other presets, also to emphasize CPU over GPU performance. In the case where there were only three presets, we chose the top preset.
Clarification of various installation questions: We installed all games using Steam or the Microsoft Store, and fully updated with the latest patches.
Motherboards: Re a Twitter comment from Cyber Cat @0xCats, "Hey @AMD Apparently according to @PrincipledTech @Intel is able to run Ryzen & Threadripper Chips on Z370 and Z390": Thanks for catching that copy/paste error in our configuration info. We made an error there. The correct processor/motherboard/BIOS version specs for the AMD procs we tested are the following: Ryzen 7 2700X/ASUS PRIME X470-PRO/4024 and Threadripper 2950X & 2990WX/ASUS PRIME X399-A/0807. We apologize for the error and will post a revised version (with changes noted) soon.
Because our goal is always to do the right thing and get the answers right, we are currently doing additional testing. We will share that data and will certainly call out if something is significantly different from what we've already published.
We are confident in our test methodology and results. We welcome questions and we are doing our best to respond to questions from our interim report, but doing so takes time. We will add responses if other issues come up.
Thanks for listening.
For almost 16 years, we have tested products for our clients because they trust our integrity. We have worked not just for any one company but for dozens of the leading technology firms, including rivals such as Intel and AMD, Microsoft and Google, Dell and HP, and many others.
Those clients trust PT in part because our integrity and our technical knowledge are beyond reproach. We work hard to be the best in both of those areas. We chose our company name to emphasize our commitments to both technology and our principles. (And, accusers saying we are only in this for the money obviously haven't read our book, Limit Your Greed!)
Before going further, we thus must categorically deny any dishonesty in our work on this project for Intel or in any of our other projects.
Now that we've gotten that off our chests, let's address the specific questions from recent videos as well as subsequent posts and tweets.
Project overview
An overview of the project will provide useful context.
Our overall goal - and Intel's specific request for this project - was to create as level a playing field as possible for comparing the AMD and Intel processors as the majority of the gaming market would likely use them. To do that, we built and configured 16 systems for this comparative testing; we had two of each processor/motherboard configuration. We matched all components where possible, the only variances being the motherboards, CPUs, and CPU coolers. (Full details are in our interim report.)
In an effort to be very transparent, we published our interim summary report on Oct 8, 2018. We will continue to be transparent and responsive to any questions.
Responses to inquiries
We have received a number of inquiries regarding the testing methodology we used and the potential for bias in favor of Intel. We are providing additional information to be as transparent as possible and to help allay these concerns.
The following list summarizes many of the inquiries we have received and our responses. (We are continuing to work on addressing additional inquiries.)
- Use of "Game Mode" on the AMD Ryzen 7 2700X: Some inquiries we have received concern the use of the Ryzen utility and the number of active cores in the AMD-based systems. Based on AMD's recommendations and our initial testing on the Threadripper processors, we found installing the AMD Ryzen Master utility and enabling the Game Mode increased most results. For consistency purposes, we did that for all AMD systems across Threadripper and Ryzen. We are now doing additional testing with the AMD systems in Creator Mode. We will update the report with the new results.
- Cooler choice: We chose Noctua for the CPU coolers, due to having almost identical systems in the NH-U14S (Intel) and NH-U14S TR4-SP3 (AMD), which allowed us to maintain a comparable thermal profile. Because we were not performing any overclocking on any configuration, and because AMD has said it was a good cooler, we stuck with the stock AMD Ryzen 7 2700X Wraith Prism cooler.
- Memory speeds: To have complete parity across all systems, and to allow the Intel Core i9 X-series and AMD Ryzen Threadripper to fully utilize memory bandwidth, we used four 16 GB DDR4 DIMMs on all configurations. We took the following memory configuration steps:
MSI Z390-A Pro motherboard (i9-9900K)
- Load Optimized BIOS defaults
- Enabled: Extreme Memory Profile (X.M.P.)
- DRAM Frequency set to DDR4-2666
- Asus Prime X299-Deluxe motherboard (i9-9900X ,i9-9980XE)
- Load Optimized BIOS defaults
- Enabled: Extreme Memory Profile (X.M.P.)
- Disabled ASUS MultiCore Enhancement to use stock Intel multicore settings
- DRAM Frequency set to DDR4-2666
- Installed Intel Turbo Boost Maxdriver/utility
- Load Optimized BIOS defaults
- Enabled: Extreme Memory Profile(X.M.P.)
- Disabled ASUS MultiCore Enhancement to use stock Intel multicore settings
- DRAM Frequency set to DDR4-2666
- Power saving & Performance mode, set to Performance
Asus Prime X399-A (Threadripper 2990WX, Threadripper 2950X)
- Load Optimized BIOS defaults
- Verify that D.O.C.P is selected for AMD-equivalent memory settings to XMP
- Performance Enhancer, set to Default
- Disabled overclocking enhancement
- DRAM frequency set to DDR4-2933
- Set Core Performance Boost to Auto
- Set performance bias to None
- Installed Ryzen Master utility
- Load Optimized BIOS defaults
- Verify that D.O.C.P is selected for AMD-equivalent memory settings to XMP
- DRAM frequency set to DDR4-2933
- Set performance bias to None
- Installed Ryzen Master utility
Quality settings: We configured all games to use the "High" or equivalent preset, versus "Ultra" or other presets, also to emphasize CPU over GPU performance. In the case where there were only three presets, we chose the top preset.
Clarification of various installation questions: We installed all games using Steam or the Microsoft Store, and fully updated with the latest patches.
Motherboards: Re a Twitter comment from Cyber Cat @0xCats, "Hey @AMD Apparently according to @PrincipledTech @Intel is able to run Ryzen & Threadripper Chips on Z370 and Z390": Thanks for catching that copy/paste error in our configuration info. We made an error there. The correct processor/motherboard/BIOS version specs for the AMD procs we tested are the following: Ryzen 7 2700X/ASUS PRIME X470-PRO/4024 and Threadripper 2950X & 2990WX/ASUS PRIME X399-A/0807. We apologize for the error and will post a revised version (with changes noted) soon.
Because our goal is always to do the right thing and get the answers right, we are currently doing additional testing. We will share that data and will certainly call out if something is significantly different from what we've already published.
We are confident in our test methodology and results. We welcome questions and we are doing our best to respond to questions from our interim report, but doing so takes time. We will add responses if other issues come up.
Thanks for listening.
41 Comments on Principled Technologies Comments on their Intel Processor Study
waiting for intel responses ..
And yet half the tech net question both hmn.
Also they made a fair few(wrong) assumptions considering their technical knowledge :p:D
And chat flip flopy crap, we went with this (res for eg) because everyone uses it , yeah guy everyone is using four sticks of ram ,wtaf.
It's like handing your dog's leash to the alcoholic, crack-addicted homeless man outside while you go into the store to do your week's grocery shopping... ...can't really turn around and say it's his fault you lost your dog! Everybody in town knows what that guy's up to...
I mean, it goes deeper, too. Intel could've caught these issues if they simply looked over the methodology. That's where the real blame is. PT makes their methodology perfectly clear. It's not like they intentionally misled or fudged results. They provided their methods, and that's what they stuck to. And that's what Intel signed up for. Putting that stuff out there was their mistake. They hired the wrong people and there's no reason why they couldn't have figured that out beforehand. Somebody slipped up.
Like man, if they had simply looked into it they'd have known before any of this was even done. So now it looks like they went in looking for skewed results. And nobody will ever know. Maybe incompetence (probably,) but if it was intentional, the incompetence excuse still flies. Some heads hit the chopping block, life goes on, and a good number of people will never know and will still buy into it. Maybe it's not dishonesty, but the end result is the same...
Semi-related, I actually really respect the degree of detail to which PT delves into when outlining their methodology. They show you exactly how they got to where they did. Same can't be said for a whole lot of benchmarkers/reviewers. They open themselves up to a lot of criticism by showing their hand. Thus far I think they've handled it well, too. If they could get it together and unify on a sensible methodology, they'd be making some really meaningful contributions. And the thing is I think they want to, but they're out of touch. I really don't think they had bad intentions, not at the moment anyway.
My question is, how do I personally get paid to do these things for Intel? Forget all the other companies, how do I become the one that sets the standard these places are using, because this isn't the first time we've seen stuff like this.
And riight? The paid study racket sounds pretty sweet. But that's the thing... ...it's a paid study done for Intel. So when the results come out in their favor it's kinda like "Well, yeah...." I don't understand the call for pitchforks there. It pretty un-apologetically and transparently is what it is.
I think some stupid choices were made. I look at the mistakes, have a laugh, and move on. This outcome is pretty much par for the course on the internet these days. Not saying it's right, but it's predictable.
I can wait for guaranteed-objective test results.
Considering the price of this CPU, objective testing is essential to me.
- They "correctly" set the Threadrippers in Game Mode, on AMD's suggestion. This is fine. This will give the best results for Threadripper. THEN they set it for Ryzen for "consistency." That's not being consistent. That is enabling a Threadripper-exclusive option for no good reason. Clearly they had no idea what Game Mode was and they never checked the Task Manager to notice that Ryzen was being effectively cut off at the knees.
- They set the XMP profiles for every motherboard, and then manually changed the frequencies. If you know anything about XMP, it's that you should not just apply a profile and then change frequencies without changing the timings too. Just stick to XMP, or stick to the whitesheet for the timings. Doing half of both led to really loose memory timings. I also question why they would set Intel to 2666 and AMD to 2933. Apparently that is the max supported frequencies for those processors, but if you have XMP profiles (and I'm sure there were several different profiles), then just choose one and don't let the motherboard set the timings. Most people will do XMP and then STOP, or do aggressive memory timing tests to go beyond.
- Choosing to do 4 memory sticks on dual-channel setups was just adding too much uncertainty. They should have done 2x8 memory kits at highest supported XMP. Anything else was just asking for this kind of complex cock up.
- Their argument of just using the AMD cooler because AMD said their cooler is good is stupid. People choose coolers irrespective of their CPU choice. Not equalizing something like this is just lazy, and definitely leads to a situation where Intel is max cooled, and AMD is maybe not hitting the highest XFR bin. Really easy problem to avoid.
Overall, I think that this group probably did a couple starter tests to get their initial benchmark setup going, and went with the one that looked reasonable but gave Intel the biggest advantage. Because Intel was the customer to please. Or, maybe they did just end up with setup because they are dumb or old school or can't be asked to do the extra research that every good PCMR member does when they build a PC.
He is very adept at using Intel made benchmarks on AMD hardware to showcase his sponsor...
But did you know that AMD has commissioned not one but thirteen reports from them as well?
Keep in mind, even offering game mode, shouldn't be done, by AMD, if it was truly detrimental.You said it yourself, that's the max supported frequency as claimed by the CPU maker. It seems that what was tested was the maximum configuration for each CPU, without overclocking. That's actually quite fair.Again, maximum possible configuration. For me, 2x8 GB is not enough, and we need either 4x 4 GB or 4x 8 GB (filling all slots, with single-rank memory, is best). Maybe another memory benchmark analysis is in order here.Meh, you could be very right here, and I agree with this part 1000%PCMR? Sigh. OK, throw yourself into a label, a stereotype, and think that's excellent. Do you have kids?
Keep in mind, I'm not really happy with this situation either, but man... what they did actually makes sense. Was it the perfectly optimized enthusiast configuration? Absolutely not. What they did was match maximum supported specifications only, and under those circumstances, AMD doesn't fare that well. Because the PC's enthusiast crowd is but a minor part of the market, doing things in this way really does make sense. It would have been nice to see overclocking performance tested as well though, and well.. AMD would look even worse then. I mean, that's what enthusiasts do, overclock, right?
Myself, I run a TR1950X in an ASRock board as my daily system. I have 7980XE and 7900X-based systems as well, but I don't use them often. I wanted to spend time with AMD's latest and see all sides since I had been using Intel for so long. There are benefits and negatives to both, and I think that this testing actually kind of showed that well.
How can you be so tone deaf? The entire purpose is to control for every possible variable, not every variable except the ones that AMD said we’re good.
I truly want to believe you you/them but your refusal to acknowledge that your methodology pales in comparison to part time youtuber’s, or that it’s a problem at all, makes that impossible.
All that said, I agree with you that it seems to be a mountain out of a mole hill. Just as you should never put all your eggs in one basket, you shouldn't rely on only a single source of information.
As for RAM, they’re again trying to have it both ways and plead ignorance “but this is what the average guy would do” while also claiming to be professionals interested in parity across systems, but without actually taking the most basic steps of controlling for every possible variable and ensuring each system is presented in the best light possible, because nobody is interested in how a Ferrari 488 performs against a Ferrari 458 in limp mode because it’s throwing codes.Does the data bring one to the conclusion that the i9 9900k is up to 50% faster “in the most likely scenario**”. No, and the EPA is a terrible reference, but they give highway miles/city mpg/and mixed, which is frankly amazing, because the EPA has actually painted a clearer picture than -anyone- and I never thought I would say that.
You don’t test “most likely” you test accurately. You test the performance potential of A vs. B, and you control for every possible variable, this is science 101, anything else is marketing at best.