Database Users

ATTN Dev's: Benchmark Standardization

Benchmark standardization would help clear up "A vs B" fight's I've seen in a many threads. I am still working on a battery life bench but here's my proposed testing standards and what I've been doing:
Tests are run from a fresh flashed phone. Software is run as it is (no change in settings). All tests are run three times with a computed average. The phone must be rebooted after each test to clear any data from the RAM. (BTW this takes fffooorrreeevvveeerrrr)
Software used: An3DBench, FPS2D, Linpack, NBench, NenaMark1 and Neocore.
Example:
ROM X Kernel X
An3DBench=3355
FPS2D=28 dev3.5
Linpack=33
NBench=2.69/4.049/0.92
NenaMark1=14.8
Neocore=25.9
Feedback and input would be greatly appreciated, then hopefully an agreed upon standard.
*Admin's, where would you like benckmark results posted? I'm sitting on 8 so far using the above methods.

Don't forget Quadrant.

Benchmarks are fun and all but in the end all that matters is how your phone runs and feels. If your happy, well that's all that matters.
Sent from my Evo CM6

+1 on quadrant
For battery life you could do two tests. One by letting the phone sit idle for like a hour or two and another by playing a movie or something for thats like 30mins or hour long. only thing that sucks is that will take forever!
I'd be willing to test a rom just pm me with witch one you want me to do.

funny part is these numbers mean NOTHING in the end it how YOUR phone run is what matters

drmacinyasha said:
Don't forget Quadrant.
Click to expand...
Click to collapse
As a few dev's have stated,and I've been able to duplicate, Quadrant numbers are easily manipulated at the kernel level. That's the reason I didn't include it in my tests.
Sporkman said:
funny part is these numbers mean NOTHING
Click to expand...
Click to collapse
Yes and no. For example An3DBench, NenaMark1 and Neocore scores mean nothing if FPS2D shows you have a high deviation. A high deviation affects gameplay, video playback, video recording and ui smoothness. I've benched a number of roms and kernels and the ones with the lowest FPS2D deviations stutter less on all areas. It's been brought up many times that the soft side needs help. As a tech going all the way back to 386's and IIe's, I've found that tests like these show where things are good and where things can be improved.

Latest TalonDEV Kernel Achievements

Just wanted to share with you guys the Quadrant Advanced Scores of Latest version of TalonDEV Kernel.
It is amazing, I've used Semaphore, CF-Root, Dark Core, Galaxian and SpeedMOD before, but never before I have seen these astonishing results, so go ahead and install it and see for yourself.
====================================================================================================================================================
EDIT
====================================================================================================================================================
<| To all the people that dismiss the scores of Quadrant benchmark as NOT being reliable or fair |>
There is a small fact that all of you guys forget when you say that the Quadrant Benchmarks Scores are not reliable and that is the fact that we all are using the same gadget, I mean you were right, if we were talking about different hardwares, but same hardware and same benchmark application with different software configurations shows you how good your choice of ROM and Kernel are, and thats it.
So while it is not reliable to compare the scores of two different phone's but IMHO it is OK to compare results with people with the same smart phone.
I completely agree with you if you argue that a phone is a phone and is not a PC that you use for 3D stuff, but higher scores in Quadrant don't mean that I can play the latest 3D game and you don't, it's just that with higher scores is probably more likely that your phone is working more smoothly than others!?
And about the battery not lasting long when you get such high scores, I have to tell you my friends, if you want your battery last longer then what in gods name are you doing here, you should have probably not even upgraded your phone to Froyo, let alone GB or even the custom ROM that you are using now.
We all want to make our phones more interesting, put more functionalities in them and make them do more and more stuff every day, and with those desires we have to sacrifice something in return, and again IMHO a little bit of battery lifetime is the least I am prepared to sacrifice for the good looks and functionalities that my phone has compared to a stock just out of the box SGS.
Besides, I have a charger that I don't even use, there is a car charger in my car, a usb cable at home near the lap top and another one attached to the desktop and I have the same situation at work, so wherever I go there is the possibility to charge my phone, I really DO NOT understand it when people talk about battery lifetime.
Good Luck to All of YOU Modders & Testers & Adventurers

it is indeed fun when you finally arrive at the kernel that just fits your phone and needs the best. Be prepared for a whole series of "flames" from those who will variously dismiss quadrant and/or your choice of kernel because it's not their choice.
Ignore them: the rom and kernel choice is always a personal preference.

1.2ghz, bigmem and no journal ext4?? Thats the only way those scores are possible lol.

shomanadeh said:
Just wanted to share with you guys the Quadrant Advanced Scores of Latest version of TalonDEV Kernel.
It is amazing, I've used Semaphore, CF-Root, Dark Core, Galaxian and SpeedMOD before, but never before I have seen these astonishing results, so go ahead and install it and see for yourself.
Click to expand...
Click to collapse
Quadrant scores aren't that reliable. Froyo 2.2.1 has lower scores than Gingerbread 2.3.4 but some Froyo roms run smoother.
Anyway, it would be nice to replicate those scores? Which Talon kernel exactly are you using? Are you on a custom rom?

This is an interesting data if you base your choice on quadrant. But with all my respect to your post: High quadrant scores just means high quadrant scores. A phone it is not PC graphic card, somebody in the android community should develop a better way to compare performance for day-to-day users.

Shinny graphics but most important for a pocket phone is smoothness on everyday usage, stability and battery consumption. Really, the Quadrant or whatever score is the least concern most people should have on choosing a kernel, be the score thrusthworthy or not.

. .

shomanadeh said:
And about the battery not lasting long when you get such high scores, I have to tell you my friends, if you want your battery last longer then what in gods name are you doing here, you should have probably not even upgraded your phone to Froyo, let alone GB or even the custom ROM that you are using now.
We all want to make our phones more interesting, put more functionalities in them and make them do more and more stuff every day, and with those desires we have to sacrifice something in return, and again IMHO a little bit of battery lifetime is the least I am prepared to sacrifice for the good looks and functionalities that my phone has compared to a stock just out of the box SGS.
Besides, I have a charger that I don't even use, there is a car charger in my car, a usb cable at home near the lap top and another one attached to the desktop and I have the same situation at work, so wherever I go there is the possibility to charge my phone, I really DO NOT understand it when people talk about battery lifetime.
Click to expand...
Click to collapse
Well, I have about 3 days with CM7 stock on medium usage. I wouldn't sacrifice that, I get out a lot and for long so battery is of prime importance, as is mobility, independant of where or how many chargers I have.
To each his own I guess, but performance numbers come low last after stability, autonomy and performance (felt) for me.

On Semaphore 1.8 quadrant can be the same 3300-3444
Sent from my GT-I9000 using xda premium

Or platypus:
http://forum.xda-developers.com/showpost.php?p=15132880&postcount=1380

I got 3910 on damian gto.
Running at stock 1000mhz on jvt
Sent from my GT-I9000 using Tapatalk

Are this scores based on statistical methods?
Possible approach:
1. Stripping down the device to a level where a minimum of interference from running apps occur. A comparable level.
2. Collect minimum 5 scores.
3. Throw out top and bottom, calculate average based on the remaining values.
If not you're like comparing how fast you can move from A to B by changing engine in your car and forget the load.
Sent from my GT-I9000 using XDA App

It's Just for FUN!!! NOT Advancing any Science Here!?!
Are this scores based on statistical methods?
Possible approach:
1. Stripping down the device to a level where a minimum of interference from running apps occur. A comparable level.
2. Collect minimum 5 scores.
3. Throw out top and bottom, calculate average based on the remaining values.
Click to expand...
Click to collapse
AAAAAAA, No, these results are not based on statistical methodes!!! and NO, non of us and not even the 150000+ people that download these test apps have tested their phone in an labratory environment nor have they created an similar environment, these are all good old home baken test results, which I dont mind at all if I may say so myself.
It's just fun to test stuff and compare results, it's not that we are measuring **** sizes or something , and we are certainly not doing it for the good of man kind or advancing science, at least I hope not.
If not you're like comparing how fast you can move from A to B by changing engine in your car and forget the load.
Click to expand...
Click to collapse
But isn't this how they do it on the race track or when we street race? Try to strip the car down and make engine more powerfull?

Bench mark
i think that benchmarch did not represent the real feel of a kernel. Take a look to Midnigth Kernel, the quadrant score in not much but the phone is really smooth
---------- Post added at 01:25 PM ---------- Previous post was at 01:25 PM ----------
i think that benchmarch did not represent the real feel of a kernel. Take a look to Midnigth Kernel, the quadrant score in not much but the phone is really smooth
---------- Post added at 01:26 PM ---------- Previous post was at 01:25 PM ----------
O-T said:
Are this scores based on statistical methods?
Possible approach:
1. Stripping down the device to a level where a minimum of interference from running apps occur. A comparable level.
2. Collect minimum 5 scores.
3. Throw out top and bottom, calculate average based on the remaining values.
If not you're like comparing how fast you can move from A to B by changing engine in your car and forget the load.
Sent from my GT-I9000 using XDA App
Click to expand...
Click to collapse
i think that benchmarch did not represent the real feel of a kernel. Take a look to Midnigth Kernel, the quadrant score in not much but the phone is really smooth

Benchmarks DO Matter!

hi. i have seen people here saying that "Benchmarks prove nothing" and even one of the Senior member saying so.
but what i have experienced after using Note and switching between different ROMS like Rocket Rom v22 and v23, Midnote, CheckRom and Stunner Rom, i have found that Yes the Benchmarks Do matter a lot in terms of just explaining how your phone will perform.
i mean i ran the bencmarks on all of these Roms and i used them all for about 4 to 5 days each, and i found that not all of these give equal results and not all of these are equal.
the Check rom performed the worst among all of these Roms. by the time i installed it, my phone started giving me issues. like the battery issue. whenever i connected the phone to the data cable and either connect to my pc or laptop or even the charger, there would a screen popup saying, battery removed! i did everything i could do to fix this issue but all in vain.
well the best Rom in terms of performance is the Rocket Rom. rocket rom gave me the highest scores and i found it to be the most stable Rom.
Midnote was ok rom. it gave me really good benchmark scores but there were lots of force closes.
i really dont want to talk about the CheckRom. it gave me the lowest scores and it proved to be the most buggy rom i have used, and then came the battery removed issue.
Surprisingly i, the ICS Stunner Rom was the Rom which gave me around 110000 score in the Browser mark in the native browser, but for the rest of the benchmarks it gave me normal scores.
so in my point of view the best rom for note for now is the Rocket rom, but if u want buttery smooth UI and 199% smoothness, go for the ICS Stunner. but the payoff is that u loose the SNote and the screen capture capability which i really love about my note.
hence proved, Benchmark Results Do make a difference!...... These are my own personal results and you may have different results but I thought I would share my findings....

aami.aami said:
hence proved, Benchmark Results Do make a difference!
Click to expand...
Click to collapse
Hardly proved! Far from it.
Benchmarks *can* give you an indication of performance, but *nothing* is better than simply trying out different ROMs and making your own objective decision.
Fwiw though, some benchmarks simply aren't worth the time and effort - Quadrant, I'm looking at you!
Regards,
Dave
Sent from my GT-N7000 using Tapatalk 2

I am gonna steal my friend's Chasmodo quote (credits to Chas)
A man without a benchmark is like a camel without electric toothbrush
end
Sent from my GT-N7000 using Tapatalk

Benchmarking could be slightly usefull when comparing different roms though optimalizations that are specifically made to reduce boottime, remove lag or whatever might not show up in te scores.
Using benchmarks to compare phones is definately useless, for example because Android forces you to run apps in the native phone resolution. The Note runs at 1280x800 which means any phone with a similar GPU but lower resolution like the standard 800x480 could synthetically outperform the Note.

Just to throw another point in there I know it can be easily disputed but take for example iOS and windows phone.
Windows phones are limited to single core and they still run smoothly due to optimization.
the latest iPhone 4s has a dual core A5s @ 800mhz, yet it still runs smoothly.
As mentioned above in the thread the only real way is to try out the phone for yourself.

Dude, it's obvious you're no scientist. You haven't proved anything trying to connect benchmark scores with stability and force closes. The only thing a benchmark could indicate is possibly smoothness. It's all about you simply trying different ROMs and finding one that is subjectively working best for you.
Sent from my superior GT-N7000 using Tapatalk

Benchmarks generally test specific capabilities of hardware: arithmetic calculations, GPU performance, etc.
That being the case they are a good yard stick for how the HARDWARE of different devices compare in terms of performance for the most part. Although software and the OS do obviously play their part, the tests are generally targeted at exercising specific aspects of the 'computer' rather than the overall OS.
OS performance is another, much more complex, aspect that benchmarks struggle to cater for. Small changes in benchmark scores between different ROMs is unlikely to tell us much. Somebody who reviews those ROMs and give a fair summary of the overall performance, stability, etc is a much better guide (like the OP, in this case - after a fashion!)

Check Rom is '****ty'?
What utter claptrap. Only proves you have no idea about how to set your phone up properly.
As for the battery issue, really? I've been running checkrom on 3 notes(mine,my wifes and my dads) and never had fcs, battery issues.
Benchmarks are for people like you.

aami.aami said:
the Check rom is the Most ****ty Rom i have used among all of these Roms. by the time i installed it, my phone started giving me ****ty issues.
Click to expand...
Click to collapse
Yo, Benchmarker, watch your language.
Calling any development '****ty' just showcases your poor manners, egomania and general absence of intelligent thinking.
The devs give you something for free.
You don't like it, move on. And keep shtum about it. That's what people with manners do.
I wish you luck in your future benchmarking endeavors, and hope never to hear about them.

Benchmarks don't prove a ****. But I do it every time I try out a new kernel and waste 5 precious minutes of my life.
By the way, calling Checkrom ****ty was insult of all checkrom devs and ultimately an insult of xda itself, we are here because of all these devs.
You simply could have said it had lowest benchmarks or even it was sub par.
May I hope that next time you will carry general code of conduct and keep your ****ty opinions within your skull ?

chasmodo said:
Yo, Benchmarker, watch your language.
Calling any development '****ty' just showcases your poor manners, egomania and general absence of intelligent thinking.
The devs give you something for free.
You don't like it, move on. And keep shtum about it. That's what people with manners do.
I wish you luck in your future benchmarking endeavors, and hope never to hear about them.
Click to expand...
Click to collapse
That's what I'm talking about.
Don't benchmark the OS and mobile, but benchmark your brain that is it capable enough to run that device at its fullest without the need of any score to prove anything.
Just my 2 cents.

aami.aami said:
hence proved, Benchmark Results Do make a difference!
Click to expand...
Click to collapse
where have you proved anything? you really are a bit deluded arent you

Benchmark are valuable in terms of relative comparison to the same benchmark.
Benchmark just don't correlate well to real world performance.
IE: Note LTE ATT on speed app runs 15-20 M's, mine runs 7-9 in same location. Same network. Yet download Engadget simultaneously and they're even.

Benchmarks prove nothing.
/wisdom

mfractal said:
Benchmarks prove nothing.
/wisdom
Click to expand...
Click to collapse
Benchmarks prove everything.
/wisdumb

Zamboney said:
Dude, it's obvious you're no scientist. You haven't proved anything trying to connect benchmark scores with stability and force closes. The only thing a benchmark could indicate is possibly smoothness. It's all about you simply trying different ROMs and finding one that is subjectively working best for you.
Sent from my superior GT-N7000 using Tapatalk
Click to expand...
Click to collapse
i really dont understand why some people dont understand what the main point some statement is. its quite simple, nothing scientific and i did not talk about any technicalities. see for your self, the more smoother the ROM is the more better the Hardware of the phone is going with the build of the ROM that's obvious, right? now that would mean more room for the hardware of the phone to be available for better benchmark results.

aami.aami said:
i really dont understand why some people dont understand what the main point some statement is. its quite simple, nothing scientific and i did not talk about any technicalities. see for your self, the more smoother the ROM is the more better the Hardware of the phone is going with the build of the ROM that's obvious, right? now that would mean more room for the hardware of the phone to be available for better benchmark results.
Click to expand...
Click to collapse
probably but not necessarily, correlation does not mean causation,
each benchmark result is moreless the weighted average of few or more individual test results - now let's take Quadrant, which is (was?) heavily biased towards 3D graphics performance - how much is that representative of ROM performance under typical (non-gaming) phone usage?
PS.
Benchmarks may matter. Size matters more.
(Anyone telling otherwise is either ashamed or trying to be nice)

From the Moderator
Watch your language....... nothing wrong with arguing about results or subjective findings, but almost all opinions are subjective..... so discuss nicely or we close the thread if flaming starts up
Thanks.... oka1

One thing apple got google and its oem beats. Dont fking look at the hardware and benchmark. Look at the user experience. Not saying ios and iphone is great for the folks who like to tinker. But for a dual core 800mhz iphone4 or ipad2 it sure runs a lot smoother than a note with stock touchwiz or some other launcher. Afterall u r using ur device, ur own judgement should tell u how good the device is not some program score. And there will always room for hardware improvement, but can the software follows? How many app is actually utilizing multi core process?
Sent from my GT-N7000 using Tapatalk 2

put new Kernels you will see big different in benchmark.
From my experience testing roms + kernels I see that benchmarks really nothing
I got with Antutu benchamrk with stock rom and stock kernel about 6900 score
but it was so laggy and battery sucks. atm using Chrack´s rom extra small + speedmod v10 and its perfect no lags good battery life smooth but the benchmark sucked got like 4500. So benchmarks aren't the way to chose a rom otherwise there is no need to test and share experience with people

[REQUEST] Antutu Benchmarking from Locked/Stock/Stock or variations

Hi. Title says it all.
If anyone has a locked bootloader, stock ROM, stock launcher, (Basically new out the box), I'd love to see your AnTuTu benchmarks.
Also, I'd be interested in any other combination of unlocked/locked bootloader, STOCK ROM, and stock or 3rd party launcher. (If not stock rom, please post your BMs and details in the threads for your ROM )
THANKS! In advance (and when you post your screen). :good:
---
https://play.google.com/store/apps/details?id=com.antutu.ABenchMark
If you're new and want to help out, (because let's face it, who else has locked/stock/stock?) what you'll do is click that link above, choose to install, send to your Nexus 7. After a couple minutes or when your N7 notifies you that AnTuTu benchmark is installed, open the device, click the 2nd tab at top called "TEST" and then hit "Start Test".
This will take about 5 minutes. You'll get a benchmark screen but go ahead and give it a few more minutes, in fact on the first benchmark screen you see, (with a score at the top) if you tap the first divider labelled "Detailed scores", you'll see that more parts of your system are still being benchmarked. Make sure to pull down this DETAILED SCORES before taking the screen screen!
After that, and all benchmarking is done, hold power+vol down to take a screen shot. You can then post from your N7 to here or send it to yourself/etc to post the screenshot here.
Attached is an example (from my N7, which is unlocked/SGT7 ROM/modified stock launcher) of what it should look like.

Locked or unlocked has zero effect on performance/becnhmarks/battery etc. Effects nothing at all.
Sent from my Nexus 7 using Tapatalk 2

Thanks for your valuable input. albundy.

Thanks for the sarcastic reply after making a thread with such a silly idea to bench with a locked and unlocked bootloader.
It makes no difference. I am curious to how you can even ponder the idea that it could to be honest.
If you where so curious about it do it yourself. Waste your own time instead of trying get a noob to waste theirs. Anybody that isn't a complete newbie knows already it makes no difference and most would even know why.

albundy2010 said:
Thanks for the sarcastic reply after making a thread with such a silly idea to bench with a locked and unlocked bootloader.
It makes no difference. I am curious to how you can even ponder the idea that it could to be honest.
If you where so curious about it do it yourself. Waste your own time instead of trying get a noob to waste theirs. Anybody that isn't a complete newbie knows already it makes no difference and most would even know why.
Click to expand...
Click to collapse
I'm looking to gather benchmarks from a variety of configurations with stock ROM - that's the goal. The bootloader is quite obviously the least important part of this. I included it because there are quite a few people who may have unlocked and rooted and don't realize that it doesn't matter and to avoid filling the thread with meaningless posts that don't at all provide to the conversation like "Does it matter if I have unlocked my bootloader?". Basically, I was providing the detail that I would love to see benchmarking off *all* configurations with stock ROM.

antutu is mostly cpu speed dependant, with some infuence on 2d/3d scores if or if not you have force gpu acceleration enabled in developer options(device main settings). mostly, scores with similar cpu speeds will score the same(root or stock) if you bench with that cpu speed locked(for example 1200mhz high/1200mhz low). if you let the device scale(set cpu speed to the highest high and lowest low) then you wont know what cpu speed its actually benchmarking because the device scales and decides what cpu speed it wants. allowing it to scale can produce a huge variation in scores and is inaccurate in providing the true scores for a certain cpu speed, stock or root.

Antutu UX I/O performance score indicator for phone "snappiness"?

Hey guys,
would you be so kind and tell me your specific antutu score from the section named above. I had now 3 nexus 6p in my hands who feel more or less snappy (same setup ofc) and it matches with their ux i/o performance score.
With that being said, mine always comes out at around 3k with my girlfriend's phone having nearly 1k more. And like I said, theirs feels much smoother when using it.
Thanks.

Benchmarks are useless for testing performance. They're supposed to be taken with a grain of salt. Run a benchmark, now run another one. And 1 more after that. Each one will lower in score every time since the CPU temp is increasing which kicks thermal throttling into gear. Besides, what's your control group? Stock, unmodified? Because you'll likely have a number of different results from people running any combination of ROMs/kernels. Some may overclock, some may disable hotplugging, using different governor or schedulers, etc. The results would be widely different and wouldn't tell you much. Aside from that, I believe there is a dedicated benchmark thread buried in this section somewhere if you want to see others results or post yours.

RoyJ said:
Benchmarks are useless for testing performance. They're supposed to be taken with a grain of salt. Run a benchmark, now run another one. And 1 more after that. Each one will lower in score every time since the CPU temp is increasing which kicks thermal throttling into gear. Besides, what's your control group? Stock, unmodified? Because you'll likely have a number of different results from people running any combination of ROMs/kernels. Some may overclock, some may disable hotplugging, using different governor or schedulers, etc. The results would be widely different and wouldn't tell you much. Aside from that, I believe there is a dedicated benchmark thread buried in this section somewhere if you want to see others results or post yours.
Click to expand...
Click to collapse
i am not talking of cpu bench results. im well aware of the thermal throttling kicking in.
i/o performance should not decrease and does not decrease over several runs.
also i stated that both mine and my girls phone were running a clean pure nexus nougat build (latest) with ex kernel and same setup.
the scores differed in points to a degree where are i can rule out benchmark tolerance.

some more ux i/o performance scores from users here, perhaps?