Database Users

ATTN Dev's: Benchmark Standardization

Benchmark standardization would help clear up "A vs B" fight's I've seen in a many threads. I am still working on a battery life bench but here's my proposed testing standards and what I've been doing:
Tests are run from a fresh flashed phone. Software is run as it is (no change in settings). All tests are run three times with a computed average. The phone must be rebooted after each test to clear any data from the RAM. (BTW this takes fffooorrreeevvveeerrrr)
Software used: An3DBench, FPS2D, Linpack, NBench, NenaMark1 and Neocore.
Example:
ROM X Kernel X
An3DBench=3355
FPS2D=28 dev3.5
Linpack=33
NBench=2.69/4.049/0.92
NenaMark1=14.8
Neocore=25.9
Feedback and input would be greatly appreciated, then hopefully an agreed upon standard.
*Admin's, where would you like benckmark results posted? I'm sitting on 8 so far using the above methods.

Don't forget Quadrant.

Benchmarks are fun and all but in the end all that matters is how your phone runs and feels. If your happy, well that's all that matters.
Sent from my Evo CM6

+1 on quadrant
For battery life you could do two tests. One by letting the phone sit idle for like a hour or two and another by playing a movie or something for thats like 30mins or hour long. only thing that sucks is that will take forever!
I'd be willing to test a rom just pm me with witch one you want me to do.

funny part is these numbers mean NOTHING in the end it how YOUR phone run is what matters

drmacinyasha said:
Don't forget Quadrant.
Click to expand...
Click to collapse
As a few dev's have stated,and I've been able to duplicate, Quadrant numbers are easily manipulated at the kernel level. That's the reason I didn't include it in my tests.
Sporkman said:
funny part is these numbers mean NOTHING
Click to expand...
Click to collapse
Yes and no. For example An3DBench, NenaMark1 and Neocore scores mean nothing if FPS2D shows you have a high deviation. A high deviation affects gameplay, video playback, video recording and ui smoothness. I've benched a number of roms and kernels and the ones with the lowest FPS2D deviations stutter less on all areas. It's been brought up many times that the soft side needs help. As a tech going all the way back to 386's and IIe's, I've found that tests like these show where things are good and where things can be improved.

Quadrant/Neocore scores based on ROM/Kernel/SetCPU/SDCard

Just thought about gathering up some info on Quadrant scores and possibly correlate that to ROM/Kernel/MicroSD setup in order to figure out the best combination. I understand that Quadrant scores are somewhat subjective and Kernel code along with HD2 Android Builds are constantly changing, but it would still be interesting to see the tendencies in the performance.
Here is my most recent result:
Quadrant Score: 2046
Neocore Score: 31.8fps
ROM: [Build][21.10.2010][NexusHD2-FRG83 V1.5 PPP + RMNET (Froyo 2.2.1)]
Kernel: Hastarin 7.7
SetCPU: 1152/1152/ondemand
MicroSD: Transcend 8GB Class 6
Please note that I ran Quadrant and Neocore multiple times and I am showing the highest score achieved. I recommend you to do the same. I have seen 1994, 1997, 2015, 2017.
Here is Quadrant Score breakdown:
CPU - 6405
Memory - 1139
I/O - 1958
2D - 309
3D - 418
I noticed that 2D and 3D scores are not fluctuating as much as the other scores, though overclocking does slightly increase their values.
Thanks,
-CJ

Here is my most recent result:
Quadrant Score: 1570
ROM: Mdeejay evo sense 2.2 rEVOlution
Kernel: Hastarin 7.7
SetCPU: 1152/1152/interactive
MicroSD: Transcend 8GB Class 6

Here is my most recent result:
Quadrant Score: 2350
ROM: Mdeejay evo sense 2.3 rEVOlution
Kernel: Hastarin 7.7
SetCPU: 1152/1152/interactive
MicroSD: Kingston 4GB Class 4

Yosef said:
Here is my most recent result:
Quadrant Score: 2350
ROM: Mdeejay evo sense 2.3 rEVOlution
Kernel: Hastarin 7.7
SetCPU: 1152/1152/interactive
MicroSD: Kingston 4GB Class 4
Click to expand...
Click to collapse
that's quite some result!
Quadrant Score: 1500-1700 (varies quite a lot)
ROM: Mdeejay FroYo HD v.3.7
Kernel: Hastarin 7.7
SetCPU: 250/1152/conservative (performance doesn't change score)
MicroSD: Sandisk Class 6 (Ultra version)
I think it would be useful for those who have quadrant advanced to post the breakdown of their score to see exactly how everyone differs so much.

Not to be a buzzkill but a synthetic benchmark result from any of these programs doesn't really mean anything. It's more than possible to have a high linpack/neocore/xyz benchmark result and have a sluggish, unresponsive UI. You can do all sorts to Android to inflate your score but what counts is being able to use the phone day to day, with no problems.

how are you guys running neocore. my phone keeps telling me i dont support qualcomm accelerated graphics. then it force closes.
ive tried nearly all android builds, with a variety of available kernels.
none of them seem to allow me to run neocore. however i can get a 2k score in quadrant advanced.

removed.

malaeus said:
how are you guys running neocore. my phone keeps telling me i dont support qualcomm accelerated graphics. then it force closes.
ive tried nearly all android builds, with a variety of available kernels.
none of them seem to allow me to run neocore. however i can get a 2k score in quadrant advanced.
Click to expand...
Click to collapse
Neocore FC could be a kernel issue (and/or maybe related to stagefright). Try Hastarin's 7.7 kernel and/or make sure stagefright is enabled.

Reno_79 said:
Not to be a buzzkill but a synthetic benchmark result from any of these programs doesn't really mean anything. It's more than possible to have a high linpack/neocore/xyz benchmark result and have a sluggish, unresponsive UI. You can do all sorts to Android to inflate your score but what counts is being able to use the phone day to day, with no problems.
Click to expand...
Click to collapse
agree 100%

Reno_79 said:
Not to be a buzzkill but a synthetic benchmark result from any of these programs doesn't really mean anything. It's more than possible to have a high linpack/neocore/xyz benchmark result and have a sluggish, unresponsive UI. You can do all sorts to Android to inflate your score but what counts is being able to use the phone day to day, with no problems.
Click to expand...
Click to collapse
Read the OPs post again.
Sent from my HTC HD2 using XDA App

Post Your Quadrant Benchmark Scores! (HD2)

Hey everyone. Please Benchmark using Quadrant and Post your Score along with a screen shot or picture of your score.
Also please state if you tested your phone under default stock conditions, or if you rooted and tweaked the heck out of it to get the high score you got.
You can download Quadrant. Go To Android Market and search Quadrant.
MY Score : 2799
i can't upload the pic now
1.To prevent spam to the forums, ALL new users are not permitted to post outside links in their messages. After approximately eight posts, you will be able to post outside links. Thanks for understanding!

I am running CM 6.1 Stable and getting about 2300-2350 quadrant score (setCPU scaling between 245 and 1200). What are you running to get 2800?
Edit: Had weather bug running in the BG, now getting around 2500.

That's the highest quadrant score I've EVER seen from any first gen. Snapdragon device. What do you have it overclocked to? On my mt4g (second gen snapdragon with dedicated gpu) overclocked to 1600mhz on a deodexed, zip aligned rom, I just barely go above 2800 on quadrant.
I had an HD2 and a Nexus and never came remotely close to those numbers on any rom/kernel/clock speed combination. I've messed with and flashed many Evos, overclocked within reason, that also average literally half of that.
Since your post count is 2 and youve already opened a 'bragging rights' type thread, I'm going to have to call BS on this. Sorry, I just call it how I see it. Or hey, maybe you're just the chosen one and have a magic phone.
Sent from my HTC Glacier using XDA App

I am absolutely thrilled with a score of 2500. I have been using a G1 for the past year and a half and I decided it was finally time to upgrade. My G1 was only getting about 580 with CM 6.1 (and that was an amazing score). I was debating whether to get a MyTouch 4G, G2 or an HD2 with the Android off the SD card. I was able to pick up an HD2 for $200 off craigslist and boy I do not regret it. I don't know how this phone can run Android off the SD card faster than off the nand, but I am thrilled. By the way I am using a Class 10 4 GB SD Card, if that makes much of a difference.
Only this I wish it could do is play my 720p videos without the codec getting all blocky/laggy. But that is probably asking way too much.

Is 2500 on stock clock speed? There must be some serious speed enhancements here recently, last I used a dual booted HD2 it was only in the 1300 range, same with my nexus, and two Evos. My nexus will max around 1800 with the miui rom overclocked to 1100mhz or so..Is there something special with the HD2's hardware that I'm missing or is this the norm now?
Sent from my HTC Glacier using XDA App

I am using HD2ONE 0.3.4b based on CM 6.1 Stable (CM is awesome BTW). I am using setCPU to run the phone at 1190mhz clock, but it still gets great Benchmarks without. I have been using CM for about a year now on my G1. It is built for speed and memory improvements as these were necessities to keep my G1 alive and kicking up until now. I would assume anyone who can't break the 2000 point quadrant score might want to take a look at HD2ONE and CM.
PS.
Also, this is the T-Mobile HTC HD2, so perhaps the 1gb ram helps as well.

I still don't see the point in benchmarks. I used to have a build (it shall remain nameless) that got over 2400 in quadrant but the UI itself was slow as all hell. My current build (in sig) get's 1700 ish but is smooth as melted butter. I'd rather have a smooth OS than a cooked phone as well.

I'm not sure about your unnamed os but cyanogenmod is running smooth as butter
Sent from my HTC HD2 using XDA App

Ok and yet again here we go.
Your scores are so insane high because of what apears to be bug in the benchmarking of the SD card. Your device is NOT ultra wtf pownage fast. Its nothing to boast about. Its just a glitch. I had a build that did 3100 and still the build from darkstone is 100% faster and only pulling 1600 points.
Benchmarks are fun to measure on devices that run off NAND not SD so for the love of god STOP the benchmark topics.

shuntje said:
Ok and yet again here we go.
Your scores are so insane high because of what apears to be bug in the benchmarking of the SD card. Your device is NOT ultra wtf pownage fast. Its nothing to boast about. Its just a glitch. I had a build that did 3100 and still the build from darkstone is 100% faster and only pulling 1600 points.
Benchmarks are fun to measure on devices that run off NAND not SD so for the love of god STOP the benchmark topics.
Click to expand...
Click to collapse
I knew it had to be something to do with Android running off the SD card and not NAND, those numbers are just insane for the setups. Thank you for clearing that up sir.

shuntje said:
Ok and yet again here we go.
Your scores are so insane high because of what apears to be bug in the benchmarking of the SD card. Your device is NOT ultra wtf pownage fast. Its nothing to boast about. Its just a glitch. I had a build that did 3100 and still the build from darkstone is 100% faster and only pulling 1600 points.
Benchmarks are fun to measure on devices that run off NAND not SD so for the love of god STOP the benchmark topics.
Click to expand...
Click to collapse
Well it's not just a glitch. cedesmith's asynchronous data method actually does improve i/o by a huge amount. There are a few spots where the SuperRam build feels faster than the desire_hd2 and hd2one 3.4 but it's really not that big a difference. Plus the battery drain is not as good on the SuperRam.
This is one of the builds shuntje is talking about. Cedesmith's desire_hd2, running slightly overclocked at 1.19. No other tweaks.

I do not think that this result came from a gap in the program
my phone is so fast
now in benchmark >>> my socre is 2963 !!!!
add me in facebook and see the pic
facebook.com/home.php?#!/profile.php?id=100000362542194

Uploaded with ImageShack.us

buzz killington said:
Well it's not just a glitch. cedesmith's asynchronous data method actually does improve i/o by a huge amount. There are a few spots where the SuperRam build feels faster than the desire_hd2 and hd2one 3.4 but it's really not that big a difference. Plus the battery drain is not as good on the SuperRam.
This is one of the builds shuntje is talking about. Cedesmith's desire_hd2, running slightly overclocked at 1.19. No other tweaks.
Click to expand...
Click to collapse
how did u get that high io score ?
i can barley get 5k at 1.2ghz
ish i get higher score with lower oc just 1.07ghz
overall without 2d/3d 2560
cpu 5800
io 5900
1.11ghz
overall without 2d/3d 2600
cpu 6400 io 5700

civicvx94 said:
Is 2500 on stock clock speed? There must be some serious speed enhancements here recently, last I used a dual booted HD2 it was only in the 1300 range, same with my nexus, and two Evos. My nexus will max around 1800 with the miui rom overclocked to 1100mhz or so..Is there something special with the HD2's hardware that I'm missing or is this the norm now?
Sent from my HTC Glacier using XDA App
Click to expand...
Click to collapse
I don't know the details, but recently I've noticed a bunch of people here posting quadrant scores in the 2100-2400+ range.
I score about 2000 @1.19 Ghz, but my build is quite a bit older than the new ones.
There definitely seem to have been some optimizations done in the most recent builds, especially the one by darkstone which does something like load the entire android filesystem in memory or something.
I'll have to try a newer build.

poweroutlet said:
I don't know the details, but recently I've noticed a bunch of people here posting quadrant scores in the 2100-2400+ range.
I score about 2000 @1.19 Ghz, but my build is quite a bit older than the new ones.
There definitely seem to have been some optimizations done in the most recent builds, especially the one by darkstone which does something like load the entire android filesystem in memory or something.
I'll have to try a newer build.
Click to expand...
Click to collapse
Most all the big numbers come from the i/o improvements. You'll know if you see the build uses cedesmith's initrd. Using asynchronous read/write improves performance but also has a higher chance of corrupting the data file.
Just off the top of my head, these builds all are using it: cedesmith's desire_hd2, hd2one, jdms, darkstone's superram also uses it for his build.

i am using HD2ONE ROM , latest version.. my quadrant score is 2200+..
after flushing 10-15 ROMs its first time i got 2200+
i m running Radio 2.15.50.xx

yha, benchmarks are great braging rights.
ive run roms that bench silly high, but ive also run roms that bench " slow" but in real workd performance kick the crap out of the " fast" roms.
atleast some one did a benchmark from the payed version so it breaks it all down, and shows actual usefull info...

^^^^^^^^^^^^^^^^^^^

The reason its scoring so high is because of i/o. I believe that's the wrong metric to look at because i believe i/o is dependent on what memory card you have. If you pay attention to the other scores, all of these screenshots are right on par with the nexus one 2.2+. Still awesome, but don't brag that your benchmark is uber leet, as its probably the same or close to everyone else here regardless of build.
To test my theory, load the same build on another SD card, preferably one that has a lower class (lower read and write speeds) and then run the benchmark and post results.
I will also test a build with cedesmiths initrid explained earlier and post my results
Sent from my HTC HD2 using XDA App

quadrant scores using custom roms

alright here a thread to post your scores by using non-stock roms
first off i will start
rom used:
android revolution 2.0.14
set cpu:
1228 MHZ
max: 1228800
min: 122800
scaling: on demand
Quadrant:
4 runs scores here directly after booting:
1st run : 2578
2nd run : 2465
3rd run : 2424
4th run : 2526
in depth score from the 4th run:
total : 2526
cpu: 5807
mem: 1133
I/O : 4754
2d : 323
3d : 613

3704.
Coredroid 2.5 kamma 1.9bfs kernal at 1.5ghz

(sorry for my bad english)
Yesterday I root the phone, but I can only SetCPU with lower frequency (768 MHz or 384) and I can not find a program just to turn it up (overclock) ... is there a way?

Moderator, please lock this thread
Uff, not this again.. Quadrant does not give accurate results of ROM's performance. It just does not. Then there are kernels that can "fool" Q to give better results, I mean kernels with different schedulers like BFS (brain **** scheduler). Any comparison between ROMs is pretty much useless when you use Quadrant.

jkoljo said:
Uff, not this again.. Quadrant does not give accurate results of ROM's performance. It just does not. Then there are kernels that can "fool" Q to give better results, I mean kernels with different schedulers like BFS (brain **** scheduler). Any comparison between ROMs is pretty much useless when you use Quadrant.
Click to expand...
Click to collapse
agree
they can chnge it in their custom kernels

jkoljo said:
Uff, not this again.. Quadrant does not give accurate results of ROM's performance. It just does not. Then there are kernels that can "fool" Q to give better results, I mean kernels with different schedulers like BFS (brain **** scheduler). Any comparison between ROMs is pretty much useless when you use Quadrant.
Click to expand...
Click to collapse
'a user' posted some info on why CoreDroid gives such high benchmark results with OC kernels. Total placebo effects.
@OP You judge a rom by how smoothly it runs for you, lag and stuff. When on earth are you ever going to use your phone to do something with DNA or play a game where the light dims and brightens over and over again. I mean, its fine for just testing and tuning, but real world results are more accurate.

Agree with the above posts.The only thing you can measure with Quadrant is your patience until you get what you expect!
But seriously,benchmarks like nenamark or linpack are far more accurate and represent real performance measurements.I personally run them once in a while to confirm that everything is running ok.
Sent from my Desire HD using XDA App

Leggenda said:
(sorry for my bad english)
Yesterday I root the phone, but I can only SetCPU with lower frequency (768 MHz or 384) and I can not find a program just to turn it up (overclock) ... is there a way?
Click to expand...
Click to collapse
Anyone can help me?

Leggenda said:
Anyone can help me?
Click to expand...
Click to collapse
Although you are completely off-topic...
Well,when in SetCpu tap menu,device selection and then autodetect speeds.It will ask for root,give it root rights and check the remember box in superuser's prompt.

[Benchmark] Mobile Web Browser Comparison for Android

Hi Guys,
Like most of you I own a Iconia500 and I've tested a lot of ROM available on this forum. But hard to decide wich one to go with. the processing power is not all and the fluidity of the interface and the browsing are a lot more important to me.
So I've decided to test 6 Roms available on the forum with this:
Test protocol:
Fresh install
setcpu installed and locked to 1000Mhz
reboot and launch of the test
Here are the results:
Score 974 - ROM: Minimalist 3.1
Score 961 - ROM: Taboonay 1.0.6
Score 961 - ROM: Virtuous Thrive 1.0.1
Score 960 - ROM: HoneyVillain 1.04
Score 954 - ROM: Acer Stock Rom
Score 948 - ROM: Virtuous Picasso 1.1.0
Score 945 - ROM: Lightspeed 1.0
Score 940 - ROM: Virtuous Galaxy 1.0.5
Score 848 - ROM: Virtuous Xoom 1.1.2
I hope if helps,
Peace

and stock score ?
I got 954

sanaell said:
and stock score ?
I got 954
Click to expand...
Click to collapse
Thanks, table updated.

Interesting tool, thanks for sharing. I get what you are after by comparing all ROMs at 1.0GHz, but don't forget that the more processing power you have (i.e. overclocking), the better your fluidity and browsing experience will be on any given ROM. I just ran a quick and dirty test (didn't reboot or anything) on my customized stock ROM with a 1.5GHz OC kernel with the "On Demand" governor (I used No-frills CPU control) and scored a 1222 with all bars being thicker than when running at 1.0GHz. Without overclocking I scored a 971. Even the stock kernel frequency tables are setup to be able to be OC'ed at 1.2GHz....so don't be afraid to goose that CPU a bit to get that tegra 2 performance you paid$ for

but OC = more power = less battery...

sanaell said:
but OC = more power = less battery...
Click to expand...
Click to collapse
Yes, but only when you are using it...after the web page has loaded the cpu scales frequency back down if you are setup properly. It really depends on how you use your tablet, but I don't notice much difference at all in battery life the way I use mine (web browsing with flash, email, occasional movies and games). I still have to charge it every night either way after 6-8hrs of heavy use. The screen takes far more battery than anything else...live wallpaper would likely take more battery per charge cycle than an on demand [email protected] depending on how you use your tablet. IMHO, OC'ing to 1.2GHz is a no-brainer unless you are one of those guys on the road that needs to get every minute of life out of it. If that is what you need, then the iconia probably wasn't the best tablet to buy. After you OC, it's hard to go back as flash/web pages etc. load quite a bit faster as shown by the benchmark test.

_motley said:
IMHO, OC'ing to 1.2GHz is a no-brainer
Click to expand...
Click to collapse
I disagree. I didn't notice enough of a difference to warrant the loss in battery. I rarely even play any games and the games I play already run at 30 fps without OC so why the need to OC at all?

WereCatf said:
I disagree. I didn't notice enough of a difference to warrant the loss in battery. I rarely even play any games and the games I play already run at 30 fps without OC so why the need to OC at all?
Click to expand...
Click to collapse
Battery drain would depend on which kernel and governor you used and how you use your tablet. Some kernel sources I have studied have their OC current set higher than is necessary. The one posted by Richard Trip/Roggin is very reasonable by default and you can even adjust uV by frequency to improve battery usage. The games you mention are likely using the tegra GPU/hardware, so the OC won't help you a bit there. I noticed the performance the most with flash loading on web pages. Give me a little more power on demand and I am a happy camper. If you really need that extra few minutes of battery, then I can understand. But, think about it...if you are browsing a lot, you can do more within a shorter period of time so it all comes out the same in the end.

I did a comparison @ same frequency to be a base for real comparison, you can overclock or use custom kernel, still, it is just a base

_motley said:
But, think about it...if you are browsing a lot, you can do more within a shorter period of time so it all comes out the same in the end.
Click to expand...
Click to collapse
My reading speed isn't affected by the clockspeed of my tablet, so I'll still spend the same amount of time on a website..

WereCatf said:
My reading speed isn't affected by the clockspeed of my tablet, so I'll still spend the same amount of time on a website..
Click to expand...
Click to collapse
When you are reading after it has loaded, the clock speed is the same as stock...it doesn't stay pegged at the OC speed all the time, the on-demand governor takes care of that for you

agreed but still the reaction is not instantaneous and some people just want to play with overclocking.
I found the benchmark tool interesting because it does not focus on pure force power but on user perception of the interface and after all I am a user

OP updated with a new ROM