According to a rather extensive report over at Anandtech.com, it appears that benchmarking of Android smartphones is in a rather bad situation considering that most, if not all, manufacturers cheat in most if not all benchmarks, making it quite impossible to do a review that will truly represent the actual everyday performance of the smartphone. Most manufacturers use so called "CPU/GPU optimizations" to put its smartphone above the competition.


Benchmark makers also deserve a part of the blame since they obviously have to be on their toes and update their benchmarks accordingly, something that Rightware did earlier this week when they noticed that Samsung's latest Galaxy Note 3 targets their own Basemark X benchmark.

The situation is not limited to Samsung, which was caught doing it a bit more extensively than other makers, as LG did practically the same thing with its LG G2 smartphone. The only current exception are Apple, which has no competition in iOS market so does not actually need any optimizations as well as Motorola, which is not a big competitor anyway.

While it focused mostly on AndEBench and Vellamo benchmarks in its report, Anandtech.com noted there are certainly other benchmarks that are influenced by certain manufacturer. There are also quite a few difference in GPU benchmarks that can be seen in offscreen benchmarks. While onscreen ones could have certain differences based on the screen, offscreen ones should be identical on two identical GPUs.

The situation is quite critical and if it continues, we will have benchmarks that will show how well did certain manufacturer did in optimizing its smartphone for a certain benchmark and will have nothing to do with actual performance in real life. Of course, there are still differences in drivers as well as other minor things but those should not influence benchmarks as significantly as do these optimizations.

We certainly hope that at least benchmark makers will put a bit more effort to at least try and hinder these attempts and hopefully make it too hard for smartphone manufacturers to do those optimizations, which will eventually make them forsake any future attempts.




Source: Anandtech.com.