TPU Math Check

[W1zzard]

With the fixed perf summary calculation I get RX 480 = 100%, GTX 1060 = 111.47% (not including wow)

Raw data

Gives me 111% in Excel too.

Anyone see something wrong with the method?

 

[W1zzard]

Putting WOW back in, I get RX 480 = 100%, GTX 1060 = 112.51%

 

[the54thvoid]

Maths looks fine to me. 3 titles are almost 30% faster, that's going to skew the average heavily.

 

[W1zzard]

Anyone see any problems? Feedback? Questions?

 

[marios15]

I had the same problem when i tried to explain, i think it has to do with what you want to calculate.
Sum of fps / game count VS Sum of percentage difference / game count.

I think what you need to do since the product you are reviewing is the baseline
I3=E3/C3
I4=E4/C4
etc..
I20=(SUM I3-18 / GAMECOUNT) * 100

Note that this will output the average percentage difference.


EDIT: Sorry i had the 1060 as baseline, fixed now

 

[R-T-B]

Feedback?

Yeah.

Math sucks!

EDIT: You may take my programmers license away now, sry.

 

[W1zzard]

I had the same problem when i tried to explain, i think it has to do with what you want to calculate.
Sum of fps / game count VS Sum of percentage difference / game count.

You need to average the percentage differences, otherwise you are favouring benchmarks that run high FPS.

Bench 1: 1000 FPS card 1, 1100 FPS card 2
Bench 2: 100 FPS card 1, 105 FPS card 2

1100 vs 1205. card2 = 109.5% - wrong, should be 107.5%

 

[ShurikN]

Yep that should be it

E divided by C multiplied by 100

64.7/58.1*100=111.359

58.5/59.5*100=98.319


ANd then add up all the percentages "G" and divide by the number of games.

 

[RustyKats]

How come you bench Rise of the Tomb Raider in DX12 when it's still slower than DX11? There's also no visual enhancement to be had. It should remain in DX11 benching unless DX12 is faster?

And for Hitman, it's faster in DX12, especially for AMD, they get a nice boost, but you bench it in DX11...

Then there's missing Doom Vulkan which has a big lead for AMD stuff, including that alone, would make them look better.. or rather, more accurate reflection of the current gaming scene.

 

[R-T-B]

How come you bench Rise of the Tomb Raider in DX12 when it's still slower than DX11? There's also no visual enhancement to be had. It should remain in DX11 benching unless DX12 is faster?

And for Hitman, it's faster in DX12, especially for AMD, they get a nice boost, but you bench it in DX11...

Then there's missing Doom Vulkan which has a big lead for AMD stuff, including that alone, would make them look better.. or rather, more accurate reflection of the current gaming scene.

This is sort of offtopic for a mathcheck thread.

Not that I like math, but the thread is what it is.

 

[FordGT90Concept]

100/58.1 = x/64.7
6470 = 58.1x
111.35972461273666092943201376936 = x

col G should be always be 100% or 100/C#*C#
col I should always be 100/C#*E#


WoW should be:
100/114.4 = x/146.4
14640 = 114.4x
127.97202797202797202797202797203 = x


On Hitman, Rise of the Tomb Raider, etc. should note if they were tested using DX12 or DX11.

 

[marios15]

Your original method was correct, maybe all your older review summaries were bad due to having cod twice?

 

[HTC]

Here's a thought: since you bench quite allot of games, why not remove ... say ... 2 best and 2 worst performers and then do the averages?

Kind of like several sports' performance judging, such as ski jumping, where you take the lowest and highest judge's votes out and use the rest for the scores. Since you bench so much more games then the judges of the sports, maybe 2 of each would be better, though just worst and best would be OK as well.

This would get rid of some "out of norm" results and would improve the overall averages result, in the end, no?

 

[FordGT90Concept]

I like that idea. It reduces bias where the game was clearly designed, or not, for a specific vendor. A bell curve.

 

[W1zzard]

Your original method was correct, maybe all your older review summaries were bad due to having cod twice?

that's correct. and lacking wow.

 

[marios15]

Here's a thought: since you bench quite allot of games, why not remove ... say ... 2 best and 2 worst performers and then do the averages?

Kind of like several sports' performance judging, such as ski jumping, where you take the lowest and highest judge's votes out and use the rest for the scores. Since you bench so much more games then the judges of the sports, maybe 2 of each would be better, though just worst and best would be OK as well.

This would get rid of some "out of norm" results and would improver the overall averages result, in the end, no?

This would be ok if you were only comparing 2 things or similar things, now w1z has to compare 20, some cards have more rops, some have more shaders, some have more ram, others different architecture.

 

[HTC]

This would be ok if you were only comparing 2 things or similar things, now w1z has to compare 20, some cards have more rops, some have more shaders, some have more ram, others different architecture.

Yeah ... it was a thought but while replying to FordGT90Concept, i realized it probably wouldn't be feasable so i deleted the reply right after.

Perhaps W1zzard can come up with a way for this to work? Dunno!

 

[qubit]

@W1zzard How about attaching that spreadsheet here so members can play about with it to help troubleshoot your issue?

 

[HTC]

Thinking some more about my above idea, and by removing the best and worst of the lot (battlefield 4 and COD black ops 3 respectively), i would get 112.9758 average (counting WoW).

Still quite similar to W1zzard's average.

EDIT

If i went and took off the 2nd best and 2nd worst of the lot (anno 2205 and just cause 3 respectively), the average would go to 112.6379: again, quite similar to W1zzard's average.

 

[DeathtoGnomes]

Here I thought this thread was about the other math, ya know, 1+1=3...

so where is Rift or DDO or even EQ2? They're all missing!

 

[raghu78]

With the fixed perf summary calculation I get RX 480 = 100%, GTX 1060 = 111.47% (not including wow)

Raw data

Gives me 111% in Excel too.

Anyone see something wrong with the method?

wizzard I think you should evaluate the use of geometric mean for average calculations as it might give the more accurate relationship between the 2 cards. Just to give an example. Here is what a post on anandtech highlighted about the problem with arithmetic mean.

http://forums.anandtech.com/showpost.php?p=38373763&postcount=1534

Assume we have benchmarks of 2 games:
1) RX480: 50 fps GTX1060: 100fps
2) RX480: 100fps GTX1060: 50 fps

Intuitively we would rate both cards equal, right?

Now lets look how TPU calculates:

RX480 GTX1060 100*GTX/RX
50 100 200
100 50 50

arith mean: 125
geom mean: 100

According to arithmetic mean the GTX 1060 is 25% faster whereas with geometric mean the cards are equal. Do some research and see if the geometric mean provides the better relationship measure.

 

[mejobloggs]

RX480 GTX1060 100*GTX/RX
50 100 200
100 50 50

arith mean: 125
geom mean: 100

According to arithmetic mean the GTX 1060 is 25% faster whereas with geometric mean the cards are equal. Do some research and see if the geometric mean provides the better relationship measure.

Maybe it'd be good if the final score favored a GPU that performed better in more games though?

In your example numbers above I see the 1060 as more favourable to me because in 2/3 games I get better performance

I think final score should reflect that

Can't edit for some reason.

I meant to finish off with:
I think final score should reflect that
Perhaps the final score could reflect that, e.g give the 1060 a final score of 112.5 percent
A combination of both arith and geom

 

[laszlo]

quite interesting; the average of % isn't reflected in average of fps %:

calculating average of averages usually give incorrect data and weighted average is the best to use...

 

[little cat]

The calculations are accurate but somethin is missing : testing Total War : Warhammer

Just to include this kind of game with mass scenes of hundreds people

 

[oZ65]

With the fixed perf summary calculation I get RX 480 = 100%, GTX 1060 = 111.47% (not including wow)

Raw data

Gives me 111% in Excel too.

Anyone see something wrong with the method?

There is an inherent error in this method, as another user has pointed out.

You should really use the geometric mean to have consistent results.