UL Benchmarks Kicks Huawei Devices from its Database over Cheating

[TheGuruStud]

What about VW? They paid cause they were first to get caught. Rest didn't. Add to that, how much has GM been given by the same government that caught VW (BTW, it wasn't Germany. It was US) in terms of handouts just to keep it alive? How much has GM paid for cheating? US hates competition. They love control. To obtain control is to blame others of evil doings that they themselves do. They will go as far as bomb and kill people over it too!
Just pointing out hypocrisy. That is all.

Just since GM bankruptcy (which was finally triggered to avoid a multi-hundred-million if not 1B+ dollar lawsuit over intentionally killing ppl), GM has used 8.9 billion in collusion with the govt. I'd be surprised if anything substantial has been paid. All of that crap that was spouted on TV was literal fake news. GM used 2 billion in escrow that was previously given to them by the govt to "pay back" part of the loan.

Meanwhile, Toyota was fined and forced to do a recall for a problem THAT NEVER EXISTED. US citizens are so dumb they don't know which pedal is brake/accel. But GM execs are not in prison and morons are still buying junk GMs that can sometimes make it out of warranty before blowing up. USA USA USA

 

[R-T-B]

I am talking about how Apple get high score. Apple clearly optimize(like not running full load, less quality) for benchmark so that they look powerful than others(Benifit of close source).

Closed source has nothing to do with it, because the UL labs has the benchmark source. If apple was cheating they'd be caught using the same "dummy test" techniques.

 

[HTC]

That was apples (no pun intended) and oranges to what Huawei did. Huawei is directly manipulating benchmarks through cheating.

In an effort to make it look like the phones in question are faster than they actually are.

Apple had code that slowed phones after years to try to extend the battery. Totally different since Apple wasn't manipulating benchmarks to try to influence purchasing decisions. And more importantly there are very practical reasons to try to reduce battery usage on older phones, as anyone who's ever had a phone with a battery over a year or two old knows. They took some heat for the design decision, but that's not comparable to outright fraud. And I would call inserting code to specifically detect 3dMark running and then oc'ing to the hilt (along with other steps) to manipulate benchmark results as crooked as it comes. Maybe it's a good thing they are on the "bad" list from a US legal standpoint, since sounds like they need to stay on it...

That's the excuse they gave. Regardless of the veracity of this excuse, the fact of the matter is they deliberately slowed down older phones and ... coincidentely ... these phones just so happened to become much slower than the newer phones when the difference wasn't anywhere near what was being portrayed. I call that deception, if not outright fraud.

Funny thing, and although both of these companies went about it in very different ways, the end result was the same: both tried to show the newer products were better than they actually were.

 

[IceShroom]

Closed source has nothing to do with it, because the UL labs has the benchmark source. If apple was cheating they'd be caught using the same "dummy test" techniques.

Can you prove that they are not cheating?

 

[Aquinus]

Can you prove that they are not cheating?

Can you prove that they are? In a court of law (in the US,) the burden of proof starts with the person or entity making the accusation. You know, the whole "innocent until proven guilty" thing.

 

[remixedcat]

The namesake in this thread was also guilty of literally stealing Cisco's Internetworking Operating System source code and just changing all mentions of Cisco. Total fail and that's prolly the worst chinese knockoff ever. Worse than handbags and jackets.

 

[Makaveli]

What about ATI, an American company, who did this in the Quake III benchmark way back in the 8500 Pro days? Companies everywhere are run by unethical weasels, that's why we need free press to help keep them accountable.

This sort of benchmark gaming is going to become a massive thing in the mobile market where heat and power constraints are so much more vital. It's why I don't even bother with mobile "GPUs" (calling them GPUs is an insult to real GPUs that NVIDIA and AMD produce).

LMAO ATI was a Canadian company not American during that time frame. They are now owned by AMD who is american during the 8500pro days there were not. Secondly if you are going to quote a GPU company for cheating in benchmarks that title belongs to Nvidia.

 

[The Norwegian Drone Pilot]

So when did it became cheating to just utilize the hardware 100% fully out to what the spesification is saying?

Is it cheating to utilize an engine fully out while it's being dyno tested?

Just because other apps doesn't utilize the hardware fully out doesn't mean that apps that is made SPESIFICALLY to take full advantage of the hardware is not allowed to do that.

It's like saying 'Booohooo, Spotify for Windows is BARELY using 10% of my computers fully power while Adobe Premiere Pro is taking the full advantage of the CPU, GPU and the RAM in the computer. That must be cheating for Adobe Premiere Pro to utilize the hardware fully out just because Spotify isn't doing it'.

Are you guys figuring out on how stupid that is?

Saying a benchmarking app has to run like a normal app is the dumbest argument to ever happen in the smartphones history when that's not the freaking point of benchmarking apps. A normal app and a benchmarking app is 2 totally different types of apps.

If you want to find out how a smartphone performs under normal usage, then you have to test the smartphones out yourself. Because what one person is saying about normal usage performance can be totally different from what another person sees as normal usage performance. So it's pretty much impossible for benchmarking apps to say what's considered normal usage performance.

 

[hat]

So when did it became cheating to just utilize the hardware 100% fully out to what the spesification is saying?

Is it cheating to utilize an engine fully out while it's being dyno tested?

Just because other apps doesn't utilize the hardware fully out doesn't mean that apps that is made SPESIFICALLY to take full advantage of the hardware is not allowed to do that.

It's like saying 'Booohooo, Spotify for Windows is BARELY using 10% of my computers fully power while Adobe Premiere Pro is taking the full advantage of the CPU, GPU and the RAM in the computer. That must be cheating for Adobe Premiere Pro to utilize the hardware fully out just because Spotify isn't doing it'.

Are you guys figuring out on how stupid that is?

Saying a benchmarking app has to run like a normal app is the dumbest argument to ever happen in the smartphones history when that's not the freaking point of benchmarking apps. A normal app and a benchmarking app is 2 totally different types of apps.

If you want to find out how a smartphone perfrms under normal usage, then you have to test the smartphones out yourself. Because what one person is saying about normal usage performance can be totally different from what another person sees as normal usage performance. So it's pretty much impossible for benchmarking apps to say what's considered normal usage performance.

No... they were overclocking the phones when it detected the benchmark running. The phone doesn't normally run like that. To use a car analogy as you did, what would be like somebody testing a car's performance when the engine is running at 10k RPM, but when you, the user, drives the car, the engine never goes above 7k RPM... therefore you were sold a car that doesn't perform as well as advertised. If you were looking for a high performance phone, you may be led to buy one of these models based on (false) performance ratings given by the benchmark... during which the phone runs faster than it normally ever would.

Benchmarks are (somewhat) a tool that we use to gauge performance of a product. It's a major part of how video card reviews are done here at TPU, by running actual benchmarking programs like 3DMARK on the card, as well as benchmarking it across a number of popular games. The results show the card's performance. What if the card was being overclocked during the review (and nobody said it was overclocked, or provided stock figures)? That would lead the reader to believe the card is faster than it actually is. In fact, a similar thing happened here on TPU, the manufacturer was sending us cards that were running faster than retail models. they mad

 

[The Norwegian Drone Pilot]

No... they were overclocking the phones when it detected the benchmark running. The phone doesn't normally run like that. To use a car analogy as you did, what would be like somebody testing a car's performance when the engine is running at 10k RPM, but when you, the user, drives the car, the engine never goes above 7k RPM... therefore you were sold a car that doesn't perform as well as advertised. If you were looking for a high performance phone, you may be led to buy one of these models based on (false) performance ratings given by the benchmark... during which the phone runs faster than it normally ever would.

Here you are totally wrong. Take the thing with Samsung earlier. Normal apps would run the GPU at 480 MHz while benchmarking apps would run it at 533 MHz. The thing here that you aren't realizing here is that it's written down in the specification that this specific Samsung phone can run the GPU at 533 MHz, out of the box. Just because normal apps doesn't use the full potential of the GPU, doesn't mean benchmarking apps can't do it. Benchmarking apps are specifically made to take full advantage of the smarthpnes highest speeds to give a score on how powerful they are.

And you are calling this cheating and overclocking?

Overclocking is going over the speeds that is written down for the phones in the specs list.

The same applies to dyno testing engines. When you dyno tests a car engine, then you pushes it to it's highest RPM number possible. If you are doing the same on normal traffic, then you wont get any different RPM while being there. You get the same limit on how high up you can take the RPM on the engine. It doesn't matter where you are.

Benchmarks are (somewhat) a tool that we use to gauge performance of a product. It's a major part of how video card reviews are done here at TPU, by running actual benchmarking programs like 3DMARK on the card, as well as benchmarking it across a number of popular games. The results show the card's performance. What if the card was being overclocked during the review (and nobody said it was overclocked, or provided stock figures)? That would lead the reader to believe the card is faster than it actually is. In fact, a similar thing happened here on TPU, the manufacturer was sending us cards that were running faster than retail models. they mad

And what do you think happens with the GPU's on computers while they are running 3DMark?

Yes, they get's up to 100% usage and the higest possible clock speeds that are written down in the specifications. Do you see the GPU only uses 70% of the power the GPU can deliver when running the latest 3DMark?

No you don't. 3DMark that is a benchmarking program is specifically made to push the GPU's to the limit of what the specs are saying it can deliver. So why should it be any different on smartphones?

Lastly. The thing with vendors giving out overcklocked GPU's to testers is not the same as utilizing the clock speeds and power of the genuine hardware fully out to what the specifications says it has.

 

[hat]

Here you are totally wrong. Take the thing with Samsung earlier. Normal apps would run the GPU at 480 MHz while benchmarking apps would run it at 533 MHz. The thing here that you aren't realizing here is that it's written down in the specification that this specific Samsung phone can run the GPU at 533 MHz, out of the box. Just because normal apps doesn't use the full potential of the GPU, doesn't mean benchmarking apps can't do it. Benchmarking apps are specifically made to take full advantage of the smarthpnes highest speeds to give a score on how powerful they are.

And you are calling this cheating and overclocking?

Overclocking is going over the speeds that is written down for the phones in the specs list.

The same applies to dyno testing engines. When you dyno tests a car engine, then you pushes it to it's highest RPM number possible. If you are doing the same on normal traffic, then you wont get any different RPM while being there. You get the same limit on how high up you can take the RPM on the engine. It doesn't matter where you are.

It sounds like you need to re-read the OP...

Benchmarks de-listed several popular Huawei devices from its database over proof of cheating in its benchmarks. Over the month, it was found that several of Huawei's devices, such as P20 Pro, Nova 3, and Play; overclocked their SoCs while ignoring all power and thermal limits, to achieve high benchmark scores, when it detected that a popular benchmark such as 3DMark, was being run. To bust this, UL Benchmarks tested the three devices with "cloaked" benchmarks, or "private benchmarks" as they call it. These apps are identical in almost every way to 3DMark, but lack the identification or branding that lets Huawei devices know when to overclock themselves to cheat the test.

I mean, it doesn't get much clearer than that. They push their phone beyond spec when it detects a benchmark is being run. Nobody's phone comes overclocked and ignoring power and thermal limits out of the box.

And what do you think happens with the GPU's on computers while they are running 3DMark?

Yes, they get's up to 100% usage and the higest possible clock speeds that are written down in the specifications. Do you see the GPU only uses 70% of the power the GPU can deliver when running the latest 3DMark?

No you don't. 3DMark that is a benchmarking program is specifically made to push the GPU's to the limit of what the specs are saying it can deliver. So why should it be any different on smartphones?

Notwithstanding some of the shadiness that happens with PC benchmarking (I remember something about PhysX)... the graphics card does not overclock itself outside of normal specifications. Sure GPU Boost happens, but that also happens under any type of load. There is nothing that says the graphics card will ignore built in power and temp limits and overclock itself well outside the range of what GPU Boost normally does when it detects 3DMark. What Huawei did, in contrast, did indeed ignore temp and power limits, and overclock the SoC, which means running the phone well outside spec to achieve a higher score. That's not pushing the hardware to 100%, in this case it's more like 135-136%. If this article were instead about fraudulent benchmarking of a graphics card, that would be like flashing the card with one of those unlocked BIOS that has temp/power/etc limits removed or pushed to obsene levels, clocking the piss out of the card, then putting that result up and making no mention of what was done, giving the illusion of better performance when no card runs like that out of the box.

Lastly. The thing with vendors giving out overcklocked GPU's to testers is not the same as utilizing the clock speeds and power of the genuine hardware fully out to what the specifications says it has.

True, but that's not what happened here. Huawei deliberately overclocked the SoC well outside spec to produce those results... so in reality the thing I linked to and this are pretty closely related to eachother. In fact, what I linked to is marginally more honest, because that performance level was accessible to normal users by installing the software that shipped with the card and selecting OC mode or gaming mode or something like that... which, being an option delivered by the manufacturer, is still within spec. What Huawei did was push the phone well outside spec, and that level of performance isn't even accessible to users (without installing 3rd party overclocking apps and whatever else it may take to get there) outside of benchmarks Huawei "tweaked" the phone for.

 

[FordGT90Concept]

And I also know Volkswagen & co. were happily continuing the practice too.

They were forced off the road. It wasn't that they defeated the test, it's that they broke the law which has an up to $40,000 fine per violating vehicle. It ended up costing Volkswagen a lot more than $8 billion too (more like $63 billion, amazing they haven't gone under).

Huawei didn't break the law like Volkswagen did. They broke UL's policy. ISPs are frequently caught doing the same when they detect a speed test running.

UL needs to try harder to obfuscate its tests.

 

[hat]

They broke UL's policy. ISPs are frequently caught doing the same when they detect a speed test running.

UL needs to try harder to obfuscate its tests.

Probably a pretty hard thing to do?

 

[Aquinus]

And you are calling this cheating and overclocking?

Overclocking is going over the speeds that is written down for the phones in the specs list.

Lets say you have a car and it's turbocharged and it was aware of when performance was being checked via a dyno or some other test and it added a little boost and ran a little richer than under normal circumstances. It makes it look more powerful but, it's not running in the same state as it was when things like emissions and gas mileage were tested but, this is like acting as if those things are still exactly the same. It's deceptive and that's a problem.

 

[FordGT90Concept]

Probably a pretty hard thing to do?

I think it's more that Google and Apple make it a hard thing to do.

The simplist solution is to record metrics of the system throughout the test. If clockspeeds go wild then those testing the system know the manufacturer is cheating it. They could also make the test very long to run so that if the system can't handle the clocks the clocks for extended times, it will be forced to thermal throttle which the metrics can catch.

 

[btarunr]

A free performance mode they didn't have before?

Not good enough. They sold those phones on false pretense, and were never going to give them that performance mode until caught cheating. At least that won't fly in EU courts.

 

[R-T-B]

Not good enough. They sold those phones on false pretense, and were never going to give them that performance mode until caught cheating. At least that won't fly in EU courts.

I never said it was good enough. Just stating that they didn't do nothing at all, even if it's a woefully inadequate reaction basically to try and save face.

Can you prove that they are not cheating?

No, but can you prove aliens aren't making all the phones phone run faster than they should, thus making all of us cheat?

CAN YOU?

 

[DeathtoGnomes]

I never said it was good enough. Just stating that they didn't do nothing at all, even if it's a woefully inadequate reaction basically to try and save face.



No, but can you prove aliens aren't making all the phones phone run faster than they should, thus making all of us cheat?

CAN YOU?

hold on putting my tinfoil hat on. cuz I dont need cheaters making us cheat.

What I dont get is how so many people dont understand the implications here. You might as well just ask them for their wallet and walk away.

 

[Vayra86]

I never said it was good enough. Just stating that they didn't do nothing at all, even if it's a woefully inadequate reaction basically to try and save face.



No, but can you prove aliens aren't making all the phones phone run faster than they should, thus making all of us cheat?

CAN YOU?

They were forced off the road. It wasn't that they defeated the test, it's that they broke the law which has an up to $40,000 fine per violating vehicle. It ended up costing Volkswagen a lot more than $8 billion too (more like $63 billion, amazing they haven't gone under).

Huawei didn't break the law like Volkswagen did. They broke UL's policy. ISPs are frequently caught doing the same when they detect a speed test running.

UL needs to try harder to obfuscate its tests.

Heh. In the EU it seems things went a bit different... Funny how that works.

 

[The Norwegian Drone Pilot]

It sounds like you need to re-read the OP...

I mean, it doesn't get much clearer than that. They push their phone beyond spec when it detects a benchmark is being run. Nobody's phone comes overclocked and ignoring power and thermal limits out of the box.

Notwithstanding some of the shadiness that happens with PC benchmarking (I remember something about PhysX)... the graphics card does not overclock itself outside of normal specifications. Sure GPU Boost happens, but that also happens under any type of load. There is nothing that says the graphics card will ignore built in power and temp limits and overclock itself well outside the range of what GPU Boost normally does when it detects 3DMark. What Huawei did, in contrast, did indeed ignore temp and power limits, and overclock the SoC, which means running the phone well outside spec to achieve a higher score. That's not pushing the hardware to 100%, in this case it's more like 135-136%. If this article were instead about fraudulent benchmarking of a graphics card, that would be like flashing the card with one of those unlocked BIOS that has temp/power/etc limits removed or pushed to obsene levels, clocking the piss out of the card, then putting that result up and making no mention of what was done, giving the illusion of better performance when no card runs like that out of the box.

True, but that's not what happened here. Huawei deliberately overclocked the SoC well outside spec to produce those results... so in reality the thing I linked to and this are pretty closely related to eachother. In fact, what I linked to is marginally more honest, because that performance level was accessible to normal users by installing the software that shipped with the card and selecting OC mode or gaming mode or something like that... which, being an option delivered by the manufacturer, is still within spec. What Huawei did was push the phone well outside spec, and that level of performance isn't even accessible to users (without installing 3rd party overclocking apps and whatever else it may take to get there) outside of benchmarks Huawei "tweaked" the phone for.

Alrighty. I might have missed that specific part that Huawei was actually overclocking their phones, which ofc is not fine. So if that's the case, then that should be criticised. However, Samsung was still falsely accused to cheating to just utilize the clock speeds the CPU and GPU does have on the specs list and out of the box. Because this wasn't overclocking anything beyond the listed clock speeds for those.

If the clock speeds was going over those listed specs, then Samsung would have cheated as well. But they haven't done that so far.

 

[hat]

Can't really comment on the Samsung case, as I don't know of it. Still, if 533MHz was advertised as normal operating range, then it should always run at 533MHz under load, not just under a benchmark.

 

[The Norwegian Drone Pilot]

Can't really comment on the Samsung case, as I don't know of it. Still, if 533MHz was advertised as normal operating range, then it should always run at 533MHz under load, not just under a benchmark.

Well, do you see Spotify for Windows run at the CPU's full speed under usage?

No you don't. Because that program is not made to run at those speeds. However, benchmarking apps is made to do so. Just because other apps / programs doesn't utilizes the hardware fully out doesn't mean other apps / programs that is made to do so can't run the hardware at full speed. Putting every programs / apps under the same boat is not a good idea.

 

[hat]

If I install a game, or some other demanding application on such a phone, I would expect it to run at 533MHz while running that application, as it did in the benchmark. I wouldn't expect it to run as such while browsing the web or checking my email. There's a difference between power saving features that scale performance to demand (which is a great thing) and having some extreme performance level that is only used while benchmarking. Creating a special performance level for benchmarking applications, which are used to (roughly) determine the performance of a part/system, is cheating and dishonest, because the user doesn't get access to that same performance level, which was used to generate the results they may be going off of to determine how well the system will perform for them. Unless they are running a benchmark.