EXTREMES in testing hardware, normal budget combinations never tested

[BoggledBeagle]

GPUs are tested with the best CPU available.
CPUs are tested with the best GPU available.

Why are sensible budget combinations never tested? As a customer you simply cannot find information anywhere about how these combinations perform.

For example graphic cards with budget friendly Intel 14500 CPU. (What is the AMD price equivalent to this CPU?)

Or CPUs with RX 7800 XT or RTX 4070 (super?) ?

 

[P4-630]

Why are sensible budget combinations never tested?

You can find a lot on youtube.

 

[Bill_Bright]

GPUs are tested with the best CPU available.
CPUs are tested with the best GPU available.

Why are sensible budget combinations never tested?

Because you don't want the component under test to be influenced (bottlenecked) by the other components, skewing the test results.

And who says your arbitrary examples of combinations are "sensible"? That is purely subjective.

You don't test a Bugatti on city streets with an amateur driver and where there are red lights, pedestrians and traffic jams. Or on the freeway where there is a 70mph speed limit.

Same with testing GPUs and CPUs. You wouldn't test the CPU with 4GB of RAM and a 5400RPM hard drive. Or a GPU with a monitor that only supports VGA and 640×480 resolution @ 60Hz.

So it only makes sense to ensure the only influences on the device are those imposed by the limitations of the device itself. Nothing else. Ensuring the components under test are not being restricted or limited or influenced in any way by other components in the test system is the ONLY WAY to fairly test and compare those components with their competitors.

Plus your arbitrary component examples (Intel 14500 CPU, AMD equivalent, RX 7800 XT, RTX 4070) are just that, arbitrary examples and totally useless to the 100s of millions of users who don't have those specific devices.

What you are asking for is to compare Computer A with Computer B. Those comparisons are everywhere.

 

[Onasi]

But the results you want can be easily extrapolated. Take a recent CPU review. Look at the desired resolution that you want to use. If at that resolution the CPU you are interested in doesn’t have a significant performance difference from the top performing SKU with a 4090 used (and anything in the ballpark of 5-10% I would not consider significant and anything lower is completely irrelevant) then the performance of the GPU you are interested in is also applicable from its review. If the delta is higher, then just take the averaged FPS from the GPU you are looking at review and subtract the necessary percentage. In the best case scenario the actual performance will be even better than what you get this way if the bottleneck turns out to be less significant in practice. It’s basic math and takes pretty much no time at all. Certainly less than it would take, say, W1zz, to rebench multiple CPU and GPU test systems every time.

 

[Lew Zealand]

Youtube has enough decent reviewers to cover a lot of sensible CPU-GPU combinations. You can't get everything but testing one component while not being bottlenecked by the others is necessary to set the performance baseline. Check out RandomGamingInHD, he might cover equipment close to yours.

For example, I got a new low end GPU recently and tested it first with my 5800X3D to establish it's baseline, and now it's in its matching machine with much lower CPU resources. So I know for sure the hideous stutters in Hogwarts Legacy are the old CPU and not the GPU which performs admirably with the 5800X3D. Beforehand I might not know if the GPU was sometimes the issue (as everyone knows HL is CPU-broken) but now I know it's 100% the CPU's problem.

 

[Bill_Bright]

But the results you want can be easily extrapolated.

If the device under test is not being restricted (bottlenecked) by some other component, then I totally agree. You can easily scale down results - or just guess with some assurance of accuracy.

But if the device under test is being bottlenecked, you cannot extrapolate or scale up (or guess) with any assurance the results are accurate. So again, if the goal of the test is to see what the test subject is capable of doing, you don't want the component under test to be restricted or influenced (bottlenecked) by any other component.

Even if you know what device is creating a bottleneck and restricting the performance of the device under test, there is no way to determine how far your device would otherwise go if it had absolutely no restrictions placed on it.

 

[ir_cow]

GPU reviews are about the true potential, not what a $100 i3 will get in FPS.

CPU reviews are direct comparisons to others, so the best GPU is used to show its potential. At the same time you can also see how far back a lower CPU is on the FPS.

For example is say a Intel 14900K gets 300 fps using a 4090 on average (making numbers up). a i3 1400f gets 120 fps. That means you are wasting money buying any video card that does over 120fps, because that cpu is your bottleneck.

 

[Onasi]

@Bill_Bright
That was implied, obviously. I am not suggesting taking the results from tests that aren’t made with as many possible bottlenecks removed as possible. You will never be able to remove all of them completely, that’s implausible, and variance in test runs also obviously exists, but aforementioned extrapolation doesn’t need perfect baseline to go off on, “reasonably established as valid” is good enough in this case.

Example: W1zz could THEORETICALLY lessen the bottleneck on CPU review by using an overclocked to the gills 4090 Matrix. Or he could THEORETICALLY reduce the bottleneck on the GPU reviews by, again, OCing the hell out of the 14900K under an industrial chiller and using a golden sample chip with perfectly tuned memory. Both will be theoretically better, but in practice, what he has is enough.

Oh, @BoggledBeagle, what I described above is what ACTUAL Extreme testing would be for reviews. What W1zz does is just sane and reasonable best practices.

 

[cvaldes]

GPUs are tested with the best CPU available.
CPUs are tested with the best GPU available.

Why are sensible budget combinations never tested? As a customer you simply cannot find information anywhere about how these combinations perform.

This is intentionally done to remove bottlenecks created by lower performing support components.

Gaming at 4K is very taxing on the GPU, the CPU can generate enough frames but it's the GPU that the bottleneck. That's why this scenario is described as "GPU bound."

Gaming at 1080p on the other hand is something that a high-performance GPU can handle without issue; the graphics card can render as many frames as the CPU can throw at it. So this why a high-end GPU is used for testing CPUs.

Remember that these tests are all relative to the same type of components (CPUs, GPUs, sometimes SSDs, etc.).

Changing one variable for testing purposes is one of the key concepts of scientific methodology, something that should have been taught to you by the time you completed middle school.

 

[agent_x007]

GPUs are tested with the best CPU available.
CPUs are tested with the best GPU available.

Why are sensible budget combinations never tested? As a customer you simply cannot find information anywhere about how these combinations perform.

For example graphic cards with budget friendly Intel 14500 CPU. (What is the AMD price equivalent for this CPU?)

Or CPUs with RX 7800 XT or RTX 4070 (super?) ?

Because it's too specific budget window that can't be "generalized" to be "optimal" for everyone.
It also doesn't help that adding "a bit more" nets you 13600(k)/14600(k), or a bit less 12400(F) with different GPU.
At which point, you need to ask yourself : How many people will use such configuration ?
Don't blame reviewers, blame Intel for throwing specs that are too similar to each other in model packed stack (+ again, going cheap now is the better deal long term). Lastly, you can pretty much guessimate "best case" for performance uplift from pure clock difference between similar mid range CPUs (12400F vs. 12600), and if that difference is a lot lower than price difference... what is the point of testing faster CPU ?

Next, most reviewers don't buy things they are testing, and spending your own money and not buying "optimal" CPU for tests isn't sustainable over long period of time.
They also don't receive "inferior" product to tests, because that's bad PR/AD, so there is no good reason to make possibly negative review from both manufacturer and review side.

Well, there are OEMs in which you can weird combos, but again money to review them is an issue (unless you are GN but solid subscriber base).

In short : Buy 14500 and make a review about it if you think it's worth buying

 

[Bill_Bright]

and variance in test runs also obviously exists

Not just test runs. Some variables are impossible to eliminate. The perfect example is comparing AMD CPUs with Intel CPUs. You can use the exact same RAM, exact same graphics card and monitor, exact same keyboard and mouse, exact same drives, exact same OS and benchmark programs. But AMD and Intel CPUs run on different motherboard platforms using different chipsets. No way around that.

That was implied, obviously.

I understood. And obviously it was obvious in your head. But we cannot assume anything is obvious to everyone reading. Hence my comment.

Next, most reviewers don't buy things they are testing

Right. This is why I like Consumer Reports. They do send out "secret shoppers" who buy test units "off the shelf", just as we consumers do. And they have professionally equipped labs and real, qualified techs doing the testing. Sadly, they don't test individual computer components for us self-builders. I've written to the editors and asked them to several times to test components, but for some reason, they don't listen to me.

They also don't receive "inferior" product to tests

No but they could receive optimally tweaked products so that is always a concern when they are provided test samples from the maker.

so there is no good reason to make negative review from both manufacturer and review side.

Ummm, I don't understand this. Please clarify.

Sure there is good reason for negative reviews. We see negative reviews often where the product fails to meet ATX standards, or fails to meet 80 Plus certification requirements, or fails to meet published specs, or fails to score well against the products it competes with.

So the ultimate good reason is to inform their readers of the facts.

 

[agent_x007]

(1)Right. This is why I like Consumer Reports. They do send out "secret shoppers" who buy test units "off the shelf", just as we consumers do. And they have professionally equipped labs and real, qualified techs doing the testing. Sadly, they don't test individual computer components for us self-builders. I've written to the editors and asked them to several times to test components, but for some reason, they don't listen to me.


(2)No but they could receive optimally tweaked products so that is always a concern when they are provided test samples from the maker.

(3)Ummm, I don't understand this. Please clarify.

(4)Sure there is good reason for negative reviews. We see negative reviews often where the product fails to meet ATX standards, or fails to meet 80 Plus certification requirements, or fails to meet published specs, or fails to score well against the products it competes with.

(5)So the ultimate good reason is to inform their readers of the facts.

Ad. 1. They don't listen, because it's too boring of a topic to make video about (keep in mind, you can use simple math to figure out perf. of such procesor based on cheaper/more expensive ones data). There is quite limited number of things a locked CPUs can have "unoptimised behind your back" vs. review sample.
Ad. 2. "Inferior" wasn't about it being badly optimized. It was about : Ad. 3/4.
Ad. 4. Example : A 14500 review from GN would probably look like RTX 2080 Super one (where 12400(F)/13400(F) is 2080/1080 Ti, and 14500 is 2080 Super). You can watch that review to see if it's "positive" or "negative" one.
Does it help consumers, Yes - 100%. Does reviewer get black listed by manufacturer for doing it... probably.
Can a single individual that tests stuff a lot, afford this kind of "heat" from potential hardware supplier ?
Unlikely.
Result : No reviews of such products.
Doesn't matter how optimized it is, when market simply has better options.
That makes it inferior product (it's about environment it releases in, and not individual part quality).

In other words : If it's not halo product, it must overcome perf/price ratio wall (unless some special "features" are desirable to make it worth it regardless). Example : If a higher CPU model isn't better enough in this metric than the lower one (features be damned) - it shouldn't get a good/positive review in my opinion (regardless of how "optimized" someone says it is).

Ad. 3. "No good reason" from my point of view = Manufacturer is "unhappy" about loss of sales, and reviewer loses "trust" of manufacturer.
That's why negative reviews aren't "popular", or repeated all the time, and why I wrote they don't have "a good reason" to be made (from business point of view).
If hype or "feelings" are at stake (because someone f***ed up MASSIVELY at one company), well - that's on them.
Hypetrains come in all sizes and can be both negative and positive (can even change mid way).
I don't mention such cases since your topic isn't about them though.

Ad. 5.

So the ultimate good reason is to inform their readers of the facts.

Ultimate "good reason" for a professional reviewer is to make a living. Not everyone can make a living out of donations after all.
Unless someone "f***s up big time", making negative reviews isn't desirable from business point of view.

PS. Inform in unbiased manner if you think a topic is worth it BUT, don't force your point of view on others.

 

[Vayra86]

GPUs are tested with the best CPU available.
CPUs are tested with the best GPU available.

Why are sensible budget combinations never tested? As a customer you simply cannot find information anywhere about how these combinations perform.

For example graphic cards with budget friendly Intel 14500 CPU. (What is the AMD price equivalent for this CPU?)

Or CPUs with RX 7800 XT or RTX 4070 (super?) ?

Its generally just not rocket science to get a good idea of relative performance. Although it does require you to actually think and read about a review, how it tests, and what the tests mean. But honestly: do you really need to know the exact performance of a given combination in a specific application? How relevant is that? Nobody buys a PC for a single application. And if you do, you'd rather find some deep dive on the specific app in question. Much more relevant is the data of relative performance. How do products stack up against each other, and how do they compare in perf/$ and stuff. You're gonna buy something anyway.

What I would certainly like to see in reviews are mixed load scenarios. Different combos not so much. But the influence of background tasks on say, games running? Very interesting. Its more than just performance, really. Its also how CPUs handle scheduling; when you're hitting bandwidth or throughput bottlenecks, etc.

Also, if you want to cross compare reviews and results over time, doing so on various combinations is just virtually impossible. Every combination is a separate dataset really.

 

[BoggledBeagle]

I understand the idea behind removing all possible bottlenecks to test the performance of the product, but the unfortunate result is that in the review of 14500 for example, you will find its gaming performance in comparison with other CPUs with 4090. You will find that it performs better than CPU B. But what if it was paired with some budget appropriate card? What if the performance is nearly identical in that case? The information how 14500 performs with 4090 is useless.

I think reviewers are just lazy. They need to test more CPU/GPU combinations. I do not say all combinations, because some combinations are silly.

Beside testing mixed load scenarios, as Vayra mentioned, there is also a gaping hole between 1 and all threads loads. Take a Cinebench as an example when testing two CPUs capable of running 32 threads. Why not to test 1, 8, 16, 24 and 32 threads loads? Testing different load intensities could be useful too. You could observe performance, frequencies, power draw, efficiency, etc.

 

[cvaldes]

I understand the idea behind removing all possible bottlenecks to test the performance of the product, but the unfortunate result is that in the review of 14500 for example, you will find its gaming performance in comparison with other CPUs with 4090. You will find that it performs better than CPU B. But what if it was paired with some budget appropriate card? What if the performance is nearly identical in that case? The information how 14500 performs with 4090 is useless.

I think reviewers are just lazy. They need to test more CPU/GPU combinations. I do not say all combinations, because some combinations are silly.

Beside testing mixed load scenarios, as Vayra mentioned, there is also a gaping hole between 1 and all threads loads. Take a Cinebench as an example when testing two CPUs capable of running 32 threads. Why not to test 1, 8, 16, 24 and 32 threads loads? Testing different load intensities could be useful too. You could observe performance, frequencies, power draw, efficiency, etc.

Why?

Any review of a graphics card or CPU ends up being a binary decision: A.) I will buy it, or B.) I will not buy it.

Some reviewers do twice the testing by using one AMD build and one Intel build. How many more combinations will it take to satisfy you, Joe Dorkwad across town, some kid in Bangladesh?

And many of these reviewers are testing 10, 15, 20 suites of games, not just one title. Plus synthetic benchmarks. Plus power, heat and acoustic testing. Taking unboxing pictures. Some even deshroud the cards to inspect the PCBs.

It takes time for reviewers to do all this. How much more time should they spend on this? Will you personally contribute to their paycheck?

This is starting to smack VERY loudly of entitlement.

Look, PC component reviewers aren't your mom cooking you your favorite breakfast.



Should the TPU folks here start taking up suggestions?

"I'd like TPU to test the Dodgy Bros. Foobar 2000 XT with my SuperSkank 450 motherboard, Dorkwad CPU cooler modded with Crapola fans, in this vintage Mian-Mi case that I bought 25 years ago and modded with my Gremle chainsaw... Oh yes, I used minty fresh Crest gel toothpaste as my TIM. Thanks!"



Ahahahahahahahahahahahahaha!!!!

 

[BoggledBeagle]

Reviews are about providing useful information to readers/viewers. NOBODY will use 14500 with 4090! Yet the reviews provide such information and do not provide information about how it performs with budget appropriate GPU.

 

[W1zzard]

GPUs are tested with the best CPU available.
CPUs are tested with the best GPU available.

I've tested with realistic hardware for like a decade .. then people kept giving me shit .. "you're bottlenecking your CPU", "you are bottlenecking the GPU" ..

 

[cvaldes]

I've tested with realistic hardware for like a decade .. then people kept giving me shit .. "you're bottlenecking your CPU", "you are bottlenecking the GPU" ..

Which is what you don't want to do if you are trying to determine the maximum potential of the hardware being tested.

That's why all serious PC hardware reviewers use the same methodology these days.

Sure, someone could test GPUs in an external GPU enclosure (like a Sonnet or Razer) plugged into a Thunderbolt port on some four-year old ultrabook. That's realistic. The external GPU enclosure lops off about a third of the performance.

If OP really thinks the marketplace is missing that type of review, he should start doing them himself and find out how easy it is to monetize those reviews.

 

[Bill_Bright]

Ad. 1. They don't listen, because it's too boring of a topic to make video about

No. They don't listen because computer components are not purchases a significant number of their readers purchase. I note they do conducts reviews on entire computers (laptops and PCs), even monitors (for typical home office use) - but not graphics cards, PSUs, etc.

Doesn't matter how optimized it is, when market simply has better options.

It matters because a good review makes for good marketing/advertising fodder. The market having better options is immaterial if the market (the consumers) are not aware of those better options. We see the results of this all the time when consumers buy inferior products, often simply because they are a few cents less expensive. The bigger problem is, most consumers don't do due diligence when researching their option.

I understand the idea behind removing all possible bottlenecks to test the performance of the product, but the unfortunate result is that in the review of 14500 for example, you will find its gaming performance in comparison with other CPUs with 4090.

No, sorry. That is incorrect. It is just not true at all. Sorry, but clearly you do not understand. First, removing all possible bottlenecks levels the playing field. Therefore, if reviewing CPUs, for example, you will find the review (assuming it is otherwise, a proper review) compares the CPU's performance against ALL other CPUs, not just one or two. And that's a very good thing.

I think reviewers are just lazy.

Clearly a blanket statement, and like all blanket statement, it is wrong. Of course, some reviewers are lazy. In fact, some self-proclaimed reviewer do absolutely nothing but repeat (even copy and paste! ) the marketing hype directly from the manufacturer, then call that a review. Now that's lazy - and pathetic. It also does a disservice to the consumer as well as legitimate reviewers.

But, as cvaldes correctly notes, other reviewers are very comprehensive, evaluating not just physical (electrical) performance in terms of power demands, efficiencies, noise, heat generation, cooling capabilities and more, but also in very "mixed load" scenarios with a wide variety of programs (real world, benchmarking, and stress).

For you to suggest such reviews are not happening is, sadly, illustrating a significant lack of knowledge and understanding on your part about what's out there in terms of review sites and reviewers. That is NOT meant to be a criticism and my apologies as I recognize it sounds like it. It is just a clear as a bell observation.

Now of course, proper research requires looking at more than just one review. But that's not hard as there are more than just one out there and they are easy to find. Just enter the brand, model number and the word "review" in your favorite search engine, then weed through the dozens of results. It will not take long to figure out which reviews are legitimate and worth the time to read, and which are not. A good review site to start with is right here at TPU.

I've tested with realistic hardware for like a decade .. then people kept giving me shit .. "you're bottlenecking your CPU", "you are bottlenecking the GPU" ..

Yeah, some people love to hate. It seems they feel emboldened to do so when hiding behind a keyboard, on the other side of the Internet. I assume you have learned to give them all the attention they deserve!

 

[Solaris17]

I think reviewers are just lazy.

what? I dont see how thats the case at all. You get a definition of what could be the best in any given scenario. You can extrapolate performance from that. There are too many combinations.

Its unrealistic and testing low to mid end hardware wont fix it.

Further tweaks can be made per game, or engine if you want to be that pandantic.

If you dont have the best GPU or CPU then you are simply slower. Thats the point. You can go look at old tests on old CPUs and how they compare to new so you can make educated guesses on where you might end up landing.

There are hundreds of CPU and GPU comparisons to each other.

I think buyers are just lazy.

 

[Lew Zealand]

I understand the idea behind removing all possible bottlenecks to test the performance of the product, but the unfortunate result is that in the review of 14500 for example, you will find its gaming performance in comparison with other CPUs with 4090. You will find that it performs better than CPU B. But what if it was paired with some budget appropriate card? What if the performance is nearly identical in that case? The information how 14500 performs with 4090 is useless.

Wrong. The 14600K is an excellent performer at stock with the 4090 even at 1440p (see TPU's review) making it a reasonable match, and the 14500 will be less than 5% behind based on it's specs. An amount you'll never notice without an fps counter active. Which means it will handle Every. Other. GPU. with ease.

I think reviewers are just lazy. They need to test more CPU/GPU combinations. I do not say all combinations, because some combinations are silly.

I think you sound entitled. Reviewers are busy and do the tests they find interesting, informative, and will drive views where they're published.

Beside testing mixed load scenarios, as Vayra mentioned, there is also a gaping hole between 1 and all threads loads. Take a Cinebench as an example when testing two CPUs capable of running 32 threads. Why not to test 1, 8, 16, 24 and 32 threads loads? Testing different load intensities could be useful too. You could observe performance, frequencies, power draw, efficiency, etc.

Because who's gonna run Cinebench at half the thread count? What would be the point of buying a 32T CPU and testing only 16T? Just buy the a 16T CPU for less if that's all you need. And the 16T performance doesn't tell you about how another software title will work with 16T, only gimped Cinebench. That information does not translate to different apps so you need to test those packages as well, which reviewers do. Some like Photoshop use only a few cores max and some like CB use all of them.

Reviews are about providing useful information to readers/viewers. NOBODY will use 14500 with 4090! Yet the reviews provide such information and do not provide information about how it performs with budget appropriate GPU.

As mentioned the 14500 will be negligibly different than the 14600K at stock and therefore will match very well with every GPU less than the 4090, and still does a good job with the 4090.

 

[BoggledBeagle]

I've tested with realistic hardware for like a decade .. then people kept giving me shit .. "you're bottlenecking your CPU", "you are bottlenecking the GPU" ..

I understand that it is impossible to satisfy all readers, that it is not possible to test all CPU/GPU combinations, but I think that each review should contain some information about performance of parts that belong to each other, to give information to support realistic decision making scenarios the consumers face.

Because who's gonna run Cinebench at half the thread count?

Nobody, but you may find yourself in situation when you load the CPU with one or combination of different applications with similar intensity.

 

[P4-630]

about performance of parts that belong to each other

What "belong to each other"

Doesn't that depend on the buyers budget and what they'd like to buy?

 

[Lew Zealand]

OK: I bought an R5 5600 and later on a 6800 XT. Based on the FPS and 1% lows of both the 5600 and the 6800 XT, it was very easy to understand where the 5600 would be the bottleneck and where the 6800 XT would be delivering >95% of it's capability. I didn't need to guess, I could look at the games I play which are reviewed here and at other places and estimate. Nobody's paired the 5600 and 6800 XT in a review that I remember reading/watching yet it was pretty easy to know where the performance would lie based on FPS reviews of the 5600 with the top end GPU when it came out, and the 6800 XT with a top end CPU when it debuted.

 

[Super Firm Tofu]

I think reviewers are just lazy. They need to test more CPU/GPU combinations. I do not say all combinations, because some combinations are silly.

I just checked for you. I await your first set of reviews.