Principled Technologies Comments on their Intel Processor Study

[W1zzard]

Today, we have seen several reports that suggested Principled Technologies (PT) published misleading information in our recent study comparing Intel's gaming processors to AMD's. We apologize for our delay in responding, but it's been a busy day, and we wanted to be as thorough as possible in addressing inquiries concerning our testing. We'll address specific questions and share more detail on our methodology in a moment, but we first must respond directly to attempts to call our integrity into question.

For almost 16 years, we have tested products for our clients because they trust our integrity. We have worked not just for any one company but for dozens of the leading technology firms, including rivals such as Intel and AMD, Microsoft and Google, Dell and HP, and many others.



Those clients trust PT in part because our integrity and our technical knowledge are beyond reproach. We work hard to be the best in both of those areas. We chose our company name to emphasize our commitments to both technology and our principles. (And, accusers saying we are only in this for the money obviously haven't read our book, Limit Your Greed!)

Before going further, we thus must categorically deny any dishonesty in our work on this project for Intel or in any of our other projects.

Now that we've gotten that off our chests, let's address the specific questions from recent videos as well as subsequent posts and tweets.

Project overview
An overview of the project will provide useful context.

Our overall goal - and Intel's specific request for this project - was to create as level a playing field as possible for comparing the AMD and Intel processors as the majority of the gaming market would likely use them. To do that, we built and configured 16 systems for this comparative testing; we had two of each processor/motherboard configuration. We matched all components where possible, the only variances being the motherboards, CPUs, and CPU coolers. (Full details are in our interim report.)

In an effort to be very transparent, we published our interim summary report on Oct 8, 2018. We will continue to be transparent and responsive to any questions.

Responses to inquiries
We have received a number of inquiries regarding the testing methodology we used and the potential for bias in favor of Intel. We are providing additional information to be as transparent as possible and to help allay these concerns.

The following list summarizes many of the inquiries we have received and our responses. (We are continuing to work on addressing additional inquiries.)

• Use of "Game Mode" on the AMD Ryzen 7 2700X: Some inquiries we have received concern the use of the Ryzen utility and the number of active cores in the AMD-based systems. Based on AMD's recommendations and our initial testing on the Threadripper processors, we found installing the AMD Ryzen Master utility and enabling the Game Mode increased most results. For consistency purposes, we did that for all AMD systems across Threadripper and Ryzen. We are now doing additional testing with the AMD systems in Creator Mode. We will update the report with the new results.
• Cooler choice: We chose Noctua for the CPU coolers, due to having almost identical systems in the NH-U14S (Intel) and NH-U14S TR4-SP3 (AMD), which allowed us to maintain a comparable thermal profile. Because we were not performing any overclocking on any configuration, and because AMD has said it was a good cooler, we stuck with the stock AMD Ryzen 7 2700X Wraith Prism cooler.
• Memory speeds: To have complete parity across all systems, and to allow the Intel Core i9 X-series and AMD Ryzen Threadripper to fully utilize memory bandwidth, we used four 16 GB DDR4 DIMMs on all configurations. We took the following memory configuration steps:

Intel
MSI Z390-A Pro motherboard (i9-9900K)

• Load Optimized BIOS defaults
• Enabled: Extreme Memory Profile (X.M.P.)
• DRAM Frequency set to DDR4-2666
• Asus Prime X299-Deluxe motherboard (i9-9900X ,i9-9980XE)
• Load Optimized BIOS defaults
• Enabled: Extreme Memory Profile (X.M.P.)
• Disabled ASUS MultiCore Enhancement to use stock Intel multicore settings
• DRAM Frequency set to DDR4-2666
• Installed Intel Turbo Boost Maxdriver/utility

Asus Prime Z370-A (i7-8086K,i7-8700K)

• Load Optimized BIOS defaults
• Enabled: Extreme Memory Profile(X.M.P.)
• Disabled ASUS MultiCore Enhancement to use stock Intel multicore settings
• DRAM Frequency set to DDR4-2666
• Power saving & Performance mode, set to Performance

AMD
Asus Prime X399-A (Threadripper 2990WX, Threadripper 2950X)

• Load Optimized BIOS defaults
• Verify that D.O.C.P is selected for AMD-equivalent memory settings to XMP
• Performance Enhancer, set to Default
• Disabled overclocking enhancement
• DRAM frequency set to DDR4-2933
• Set Core Performance Boost to Auto
• Set performance bias to None
• Installed Ryzen Master utility

Asus Prime X470 Pro (Ryzen 72700X)

• Load Optimized BIOS defaults
• Verify that D.O.C.P is selected for AMD-equivalent memory settings to XMP
• DRAM frequency set to DDR4-2933
• Set performance bias to None
• Installed Ryzen Master utility

Resolution settings: One goal of this study was to test the CPUs and their graphics subsystems, not the GPUs, so we ran the tests at the most common gaming resolution (62.06%), 1920×1080, according to the Steam Hardware Survey: https://store.steampowered.com/hwsurvey/Steam-Hardware-Software- Survey-Welcome-to-Steam. This allowed us to minimize any GPU-based bottlenecks on the rendering pipeline.

Quality settings: We configured all games to use the "High" or equivalent preset, versus "Ultra" or other presets, also to emphasize CPU over GPU performance. In the case where there were only three presets, we chose the top preset.

Clarification of various installation questions: We installed all games using Steam or the Microsoft Store, and fully updated with the latest patches.

Motherboards: Re a Twitter comment from Cyber Cat @0xCats, "Hey @AMD Apparently according to @PrincipledTech @Intel is able to run Ryzen & Threadripper Chips on Z370 and Z390": Thanks for catching that copy/paste error in our configuration info. We made an error there. The correct processor/motherboard/BIOS version specs for the AMD procs we tested are the following: Ryzen 7 2700X/ASUS PRIME X470-PRO/4024 and Threadripper 2950X & 2990WX/ASUS PRIME X399-A/0807. We apologize for the error and will post a revised version (with changes noted) soon.

Because our goal is always to do the right thing and get the answers right, we are currently doing additional testing. We will share that data and will certainly call out if something is significantly different from what we've already published.

We are confident in our test methodology and results. We welcome questions and we are doing our best to respond to questions from our interim report, but doing so takes time. We will add responses if other issues come up.

Thanks for listening.

View at TechPowerUp Main Site

 

[Rahmat Sofyan]

littlebit too late .. kudos to Steve from techspot.com ..

waiting for intel responses ..

 

[DeOdView]

The question is... How can/will anyone trusted PT, again?! Note: need to be trusted first, prior to again.

 

[Mistral]

A hilarious read, thank you. Tech sites/tube will pick this apart nicely.

 

[iO]

"We're sorry we got caught."

 

[TheoneandonlyMrK]

"clients trust PT in part because our integrity and our technical knowledge are beyond reproach"

And yet half the tech net question both hmn.
Also they made a fair few(wrong) assumptions considering their technical knowledge


And chat flip flopy crap, we went with this (res for eg) because everyone uses it , yeah guy everyone is using four sticks of ram ,wtaf.

 

[Vya Domus]

How can/will anyone trusted PT

I for one have never heard of them. Doubt there was any trust to be lost in the first place in this particular area.

 

[Steevo]

They should post how much they got paid so AMD could have a fair shot at buying their morals as well, or maybe they could bid publicly so all would know how much it costs to buy them.

 

[Basard]

Blowing smoke up corporate asses for sixteen years...... is more like it.

 

[DeOdView]

I for one have never heard of them. Doubt there was any trust to be lost in the first place in this particular area.

Ahhh... you forgot to read the Highlight + Underline NOTE!

 

[Space Lynx]

If you watch Gamers Nexus Steve Burke interview PT co-founder from yesterday, the co-founder states that 64gb ram is considered normal for most users on 2700x and 9900k... lolol what a joke. yet, he has been benching computer stuff for 30+ years or so he claims... even if true, he is super out of touch with 2700k and 9900k/8700k owners.

 

[robot zombie]

LOL, I mean they trusted Principled Technologies.

It's like handing your dog's leash to the alcoholic, crack-addicted homeless man outside while you go into the store to do your week's grocery shopping... ...can't really turn around and say it's his fault you lost your dog! Everybody in town knows what that guy's up to...

I mean, it goes deeper, too. Intel could've caught these issues if they simply looked over the methodology. That's where the real blame is. PT makes their methodology perfectly clear. It's not like they intentionally misled or fudged results. They provided their methods, and that's what they stuck to. And that's what Intel signed up for. Putting that stuff out there was their mistake. They hired the wrong people and there's no reason why they couldn't have figured that out beforehand. Somebody slipped up.

Like man, if they had simply looked into it they'd have known before any of this was even done. So now it looks like they went in looking for skewed results. And nobody will ever know. Maybe incompetence (probably,) but if it was intentional, the incompetence excuse still flies. Some heads hit the chopping block, life goes on, and a good number of people will never know and will still buy into it. Maybe it's not dishonesty, but the end result is the same...

Semi-related, I actually really respect the degree of detail to which PT delves into when outlining their methodology. They show you exactly how they got to where they did. Same can't be said for a whole lot of benchmarkers/reviewers. They open themselves up to a lot of criticism by showing their hand. Thus far I think they've handled it well, too. If they could get it together and unify on a sensible methodology, they'd be making some really meaningful contributions. And the thing is I think they want to, but they're out of touch. I really don't think they had bad intentions, not at the moment anyway.

 

[cadaveca]

I think many of you are blowing this out of proportion, but yeah, there are some issues here.


My question is, how do I personally get paid to do these things for Intel? Forget all the other companies, how do I become the one that sets the standard these places are using, because this isn't the first time we've seen stuff like this.

 

[yotano211]

If you watch Gamers Nexus Steve Burke interview PT co-founder from yesterday, the co-founder states that 64gb ram is considered normal for most users on 2700x and 9900k... lolol what a joke. yet, he has been benching computer stuff for 30+ years or so he claims... even if true, he is super out of touch with 2700k and 9900k/8700k owners.

64gb??, I am still on 16gb and most people that I know in person still have 8gb. Only 2 that I know use 64gb and they are in the movie/song rendering business plus other things.

 

[robot zombie]

I think many of you are blowing this out of proportion, but yeah, there are some issues here.


My question is, how do I personally get paid to do these things for Intel? Forget all the other companies, how do I become the one that sets the standard these places are using, because this isn't the first time we've seen stuff like this.

It's one benchmark among many... ...even if it is official, you still have to compare several to get a working picture. Most of them are going to get at least one thing wrong or have something not applicable to you.

And riight? The paid study racket sounds pretty sweet. But that's the thing... ...it's a paid study done for Intel. So when the results come out in their favor it's kinda like "Well, yeah...." I don't understand the call for pitchforks there. It pretty un-apologetically and transparently is what it is.

I think some stupid choices were made. I look at the mistakes, have a laugh, and move on. This outcome is pretty much par for the course on the internet these days. Not saying it's right, but it's predictable.

 

[RealNeil]

Reviews from trusted sites are what I'm looking for.
I can wait for guaranteed-objective test results.

Considering the price of this CPU, objective testing is essential to me.

 

[Ahhzz]

If you watch Gamers Nexus Steve Burke interview PT co-founder from yesterday, the co-founder states that 64gb ram is considered normal for most users on 2700x and 9900k... lolol what a joke. yet, he has been benching computer stuff for 30+ years or so he claims... even if true, he is super out of touch with 2700k and 9900k/8700k owners.

seriously? I run 32gb, and bet I'm way outside the norm, or at least I was when I did that 3 years ago....

 

[danbert2000]

The big issues I see with this testing:

- They "correctly" set the Threadrippers in Game Mode, on AMD's suggestion. This is fine. This will give the best results for Threadripper. THEN they set it for Ryzen for "consistency." That's not being consistent. That is enabling a Threadripper-exclusive option for no good reason. Clearly they had no idea what Game Mode was and they never checked the Task Manager to notice that Ryzen was being effectively cut off at the knees.

- They set the XMP profiles for every motherboard, and then manually changed the frequencies. If you know anything about XMP, it's that you should not just apply a profile and then change frequencies without changing the timings too. Just stick to XMP, or stick to the whitesheet for the timings. Doing half of both led to really loose memory timings. I also question why they would set Intel to 2666 and AMD to 2933. Apparently that is the max supported frequencies for those processors, but if you have XMP profiles (and I'm sure there were several different profiles), then just choose one and don't let the motherboard set the timings. Most people will do XMP and then STOP, or do aggressive memory timing tests to go beyond.

- Choosing to do 4 memory sticks on dual-channel setups was just adding too much uncertainty. They should have done 2x8 memory kits at highest supported XMP. Anything else was just asking for this kind of complex cock up.

- Their argument of just using the AMD cooler because AMD said their cooler is good is stupid. People choose coolers irrespective of their CPU choice. Not equalizing something like this is just lazy, and definitely leads to a situation where Intel is max cooled, and AMD is maybe not hitting the highest XFR bin. Really easy problem to avoid.

Overall, I think that this group probably did a couple starter tests to get their initial benchmark setup going, and went with the one that looked reasonable but gave Intel the biggest advantage. Because Intel was the customer to please. Or, maybe they did just end up with setup because they are dumb or old school or can't be asked to do the extra research that every good PCMR member does when they build a PC.

 

[stimpy88]

I'm very surprised that Shrout did not get more of Intels crooked business. Probably next time...

He is very adept at using Intel made benchmarks on AMD hardware to showcase his sponsor...

 

[HTC]

Principled Technologies is sponsored by Intel and Intel is also a major developer of BenchmarkXPRT family of benchmarks, which is published by none other then Principled Technologies.

But did you know that AMD has commissioned not one but thirteen reports from them as well?

 

[cadaveca]

The big issues I see with this testing:

- They "correctly" set the Threadrippers in Game Mode, on AMD's suggestion. This is fine. This will give the best results for Threadripper. THEN they set it for Ryzen for "consistency." That's not being consistent. That is enabling a Threadripper-exclusive option for no good reason. Clearly they had no idea what Game Mode was and they never checked the Task Manager to notice that Ryzen was being effectively cut off at the knees.

Game mode with "normal" Ryzen chips is offered because there are benefits to turning SMT off due to the cache design. If it had zero benefit, it wouldn't be offered. That's all that Game Mode did at first. Disable SMT. Then Threaderripper came out, and Game Mode disabled a whole core's worth of CPU, but leave the memory access intact. So can also set memory access to be local-only, but that disables half the memory.

Keep in mind, even offering game mode, shouldn't be done, by AMD, if it was truly detrimental.

- They set the XMP profiles for every motherboard, and then manually changed the frequencies. If you know anything about XMP, it's that you should not just apply a profile and then change frequencies without changing the timings too. Just stick to XMP, or stick to the whitesheet for the timings. Doing half of both led to really loose memory timings. I also question why they would set Intel to 2666 and AMD to 2933. Apparently that is the max supported frequencies for those processors, but if you have XMP profiles (and I'm sure there were several different profiles), then just choose one and don't let the motherboard set the timings. Most people will do XMP and then STOP, or do aggressive memory timing tests to go beyond.

You said it yourself, that's the max supported frequency as claimed by the CPU maker. It seems that what was tested was the maximum configuration for each CPU, without overclocking. That's actually quite fair.

- Choosing to do 4 memory sticks on dual-channel setups was just adding too much uncertainty. They should have done 2x8 memory kits at highest supported XMP. Anything else was just asking for this kind of complex cock up.

Again, maximum possible configuration. For me, 2x8 GB is not enough, and we need either 4x 4 GB or 4x 8 GB (filling all slots, with single-rank memory, is best). Maybe another memory benchmark analysis is in order here.

- Their argument of just using the AMD cooler because AMD said their cooler is good is stupid. People choose coolers irrespective of their CPU choice. Not equalizing something like this is just lazy, and definitely leads to a situation where Intel is max cooled, and AMD is maybe not hitting the highest XFR bin. Really easy problem to avoid.

Meh, you could be very right here, and I agree with this part 1000%

Overall, I think that this group probably did a couple starter tests to get their initial benchmark setup going, and went with the one that looked reasonable but gave Intel the biggest advantage. Because Intel was the customer to please. Or, maybe they did just end up with setup because they are dumb or old school or can't be asked to do the extra research that every good PCMR member does when they build a PC.

PCMR? Sigh. OK, throw yourself into a label, a stereotype, and think that's excellent. Do you have kids?

Keep in mind, I'm not really happy with this situation either, but man... what they did actually makes sense. Was it the perfectly optimized enthusiast configuration? Absolutely not. What they did was match maximum supported specifications only, and under those circumstances, AMD doesn't fare that well. Because the PC's enthusiast crowd is but a minor part of the market, doing things in this way really does make sense. It would have been nice to see overclocking performance tested as well though, and well.. AMD would look even worse then. I mean, that's what enthusiasts do, overclock, right?

Myself, I run a TR1950X in an ASRock board as my daily system. I have 7980XE and 7900X-based systems as well, but I don't use them often. I wanted to spend time with AMD's latest and see all sides since I had been using Intel for so long. There are benefits and negatives to both, and I think that this testing actually kind of showed that well.

 

[svan71]

Much rather have cas 10, 3200

 

[Fiveohfour]

One does not simply say “we used the included box cooler because AMD SAID IT WAS A GOOD COOLER” and then in the next sentence pretend to be concerned with testing parity across systems.

How can you be so tone deaf? The entire purpose is to control for every possible variable, not every variable except the ones that AMD said we’re good.

I truly want to believe you you/them but your refusal to acknowledge that your methodology pales in comparison to part time youtuber’s, or that it’s a problem at all, makes that impossible.

 

[moproblems99]

What they did was match maximum supported specifications only

It seems to be that you would want to try and test on the most likely scenario and not the maximum or minimum. Sort of like how the EPA tests fuel economy, they don't do it idling around town or pedal to the metal. They also appear to use a drive cycle that no one on Earth replicates with any frequency as people rarely achieve the EPA results.

You said it yourself, that's the max supported frequency as claimed by the CPU maker. It seems that what was tested was the maximum configuration for each CPU, without overclocking. That's actually quite fair.

It doesn't seem likely they are incompetent since they have 30 years of experience benchmarking so you would think that they would know how the memory systems of each architecture would work to avoid the disparity of timings. And going along with that, they seemed to tweak only the things that would negatively impact the Ryzen processors which leads to me to believe they are not incompetent and knew what they were doing.

All that said, I agree with you that it seems to be a mountain out of a mole hill. Just as you should never put all your eggs in one basket, you shouldn't rely on only a single source of information.

 

[Fiveohfour]

You said it yourself, that's the max supported frequency as claimed by the CPU maker. It seems that what was tested was the maximum configuration for each CPU, without overclocking. That's actually quite fair.

They either had no idea what did what to what, or they did and that makes it even worse. as I said their methodology pales in comparison to part time amateur youtuber’s and there’s zero excuse for that. They contradict themselves claiming they understand the importance of parity and then saying but we used the stock cooler because “amd said it was good”. Come on.

As for RAM, they’re again trying to have it both ways and plead ignorance “but this is what the average guy would do” while also claiming to be professionals interested in parity across systems, but without actually taking the most basic steps of controlling for every possible variable and ensuring each system is presented in the best light possible, because nobody is interested in how a Ferrari 488 performs against a Ferrari 458 in limp mode because it’s throwing codes.

It seems to be that you would want to try and test on the most likely scenario and not the maximum or minimum. Sort of like how the EPA tests fuel economy, they don't do it idling around town or pedal to the metal. They also appear to use a drive cycle that no one on Earth replicates with any frequency as people rarely achieve the EPA results.

You don’t test “most likely” you test accurately. You test the performance potential of A vs. B, and you control for every possible variable, this is science 101, anything else is marketing at best.



It doesn't seem likely they are incompetent since they have 30 years of experience benchmarking so you would think that they would know how the memory systems of each architecture would work to avoid the disparity of timings. And going along with that, they seemed to tweak only the things that would negatively impact the Ryzen processors which leads to me to believe they are not incompetent and knew what they were doing.

All that said, I agree with you that it seems to be a mountain out of a mole hill. Just as you should never put all your eggs in one basket, you shouldn't rely on only a single source of information.

Does the data bring one to the conclusion that the i9 9900k is up to 50% faster “in the most likely scenario**”. No, and the EPA is a terrible reference, but they give highway miles/city mpg/and mixed, which is frankly amazing, because the EPA has actually painted a clearer picture than -anyone- and I never thought I would say that.

You don’t test “most likely” you test accurately. You test the performance potential of A vs. B, and you control for every possible variable, this is science 101, anything else is marketing at best.