Related
Benchmark standardization would help clear up "A vs B" fight's I've seen in a many threads. I am still working on a battery life bench but here's my proposed testing standards and what I've been doing:
Tests are run from a fresh flashed phone. Software is run as it is (no change in settings). All tests are run three times with a computed average. The phone must be rebooted after each test to clear any data from the RAM. (BTW this takes fffooorrreeevvveeerrrr)
Software used: An3DBench, FPS2D, Linpack, NBench, NenaMark1 and Neocore.
Example:
ROM X Kernel X
An3DBench=3355
FPS2D=28 dev3.5
Linpack=33
NBench=2.69/4.049/0.92
NenaMark1=14.8
Neocore=25.9
Feedback and input would be greatly appreciated, then hopefully an agreed upon standard.
*Admin's, where would you like benckmark results posted? I'm sitting on 8 so far using the above methods.
Don't forget Quadrant.
Benchmarks are fun and all but in the end all that matters is how your phone runs and feels. If your happy, well that's all that matters.
Sent from my Evo CM6
+1 on quadrant
For battery life you could do two tests. One by letting the phone sit idle for like a hour or two and another by playing a movie or something for thats like 30mins or hour long. only thing that sucks is that will take forever!
I'd be willing to test a rom just pm me with witch one you want me to do.
funny part is these numbers mean NOTHING in the end it how YOUR phone run is what matters
drmacinyasha said:
Don't forget Quadrant.
Click to expand...
Click to collapse
As a few dev's have stated,and I've been able to duplicate, Quadrant numbers are easily manipulated at the kernel level. That's the reason I didn't include it in my tests.
Sporkman said:
funny part is these numbers mean NOTHING
Click to expand...
Click to collapse
Yes and no. For example An3DBench, NenaMark1 and Neocore scores mean nothing if FPS2D shows you have a high deviation. A high deviation affects gameplay, video playback, video recording and ui smoothness. I've benched a number of roms and kernels and the ones with the lowest FPS2D deviations stutter less on all areas. It's been brought up many times that the soft side needs help. As a tech going all the way back to 386's and IIe's, I've found that tests like these show where things are good and where things can be improved.
Do you run a non-sprint ROM?
Help us out by booting your phone without SetCPU enabled and post your linpack score* on the wiki:
http://forum.xda-developers.com/wiki/index.php?title=HTC_Supersonic/ROMs
Make sure your version matches the version posted before posting your score.
On behalf of the community, thanks!
* Linpack scores are not the be-all and end-all of the value of a ROM. Results may vary from device to device. If you experience an erection lasting more than four hours, consult your physician immediately.
Why not post Quadrant scores too?
by all means, add a column for quadrant.
I just used linpack because i wanted to keep it simple. But if you want to add quadrant scores (or any other benchmark) go ahead
I like this idea Quadrant column added
Added scores for Fresh - using HTC's current kernel. Don't kernels effect a roms performance; could skew each person's results, right?
The problem with this set up si that the scores very to much from device to device and that depending on how the phone is set up etc or how the test is run determines the outcome.
We saw the same issue when neif was doing his rom comparison. some where taking averages some where taking best some where using everything stock some werent even with the test instruction unless one person does all the tests exactly the same for each rom on the same phone with the same set up .... it makes the numbers very useless in my opinion. Thats not even accounting for diffrent rom versions kernels etc.
So i would use this with a big grain of salt as there is basically no standard used in obtaining it if it is done as a community and YMMV.
omegasun18 said:
The problem with this set up si that the scores very to much from device to device and that depending on how the phone is set up etc or how the test is run determines the outcome.
We saw the same issue when neif was doing his rom comparison. some where taking averages some where taking best some where using everything stock some werent even with the test instruction unless one person does all the tests exactly the same for each rom on the same phone with the same set up .... it makes the numbers very useless in my opinion. Thats not even accounting for diffrent rom versions kernels etc.
So i would use this with a big grain of salt as there is basically no standard used in obtaining it if it is done as a community and YMMV.
Click to expand...
Click to collapse
+2 THIS for the sake of good science!!!!
Who knows how many variables you are not taking into account!
If you really want to do this right, you need to ensure that the ROM is the SINGLE VARIABLE for the tests.
Now who wants to take the time to do that???
I have never even remotely tried to update a wiki page, but if someone wants to post my score, running Fresh ROM 3.3.0.1 with the Netarchy 4.1.9.1 BFS kernel I got 35.523 Linpack.
Anyone else being unable to run Quadrant? I run Quadrant and it gets to the first graphics and after that the benchmark just cancels itself.
Yes, yes I know "benchmarks are useless". I still would like to be able to run it, though, and see what my $500+ phone can get
DarkAgent said:
Anyone else being unable to run Quadrant? I run Quadrant and it gets to the first graphics and after that the benchmark just cancels itself.
Yes, yes I know "benchmarks are useless". I still would like to be able to run it, though, and see what my $500+ phone can get
Click to expand...
Click to collapse
Quadrant Standard will not run on mine either.
Ran fine for me.
I ran it again; this time it got past the first graphics. However, it didn't finish; it canceled itself after the world.
Similarly, the graphics on the test were weird, such as the world was all gray instead of what it should be.
i got quadrant advance to run once after install but after that wouldn't work
I am using Quadrant Standard, just ran it a second time and it made it through again. I did notice that on some of the graphics tests the details are missing. For example, the planets are just grey orbs. Anyway, scored 1800 that time around.
same here
I did a restart once and was able to get through the full benchmark once, but the graphics were missing all details.
Otherwise, it usually stalls at database writing, then quits during the graphics section.
As the apps author notes in the description box of his app, the Nexus S is not yet supported as he needs to update his app to work better with Gingerbread.
Also answered this was answered in the stickied FAQ.
unremarked said:
Q: Linpack/Quadrant Scores?
Linpack doesn't play very nicely with the Galaxy S lineup generating very low numbers in comparison to HTC devices. This is because the processors that are powering these devices operate in different ways. Does this mean the Nexus S processors are slower? Not really, just different. As for Quadrant, it doesn't currently work very well with Gingerbread and needs to be updated. The scores folks have been seeing though put it in the 1600 range without a modified kernel.
Click to expand...
Click to collapse
Sent from my Nexus S using XDA App
Just wanted to share with you guys the Quadrant Advanced Scores of Latest version of TalonDEV Kernel.
It is amazing, I've used Semaphore, CF-Root, Dark Core, Galaxian and SpeedMOD before, but never before I have seen these astonishing results, so go ahead and install it and see for yourself.
====================================================================================================================================================
EDIT
====================================================================================================================================================
<| To all the people that dismiss the scores of Quadrant benchmark as NOT being reliable or fair |>
There is a small fact that all of you guys forget when you say that the Quadrant Benchmarks Scores are not reliable and that is the fact that we all are using the same gadget, I mean you were right, if we were talking about different hardwares, but same hardware and same benchmark application with different software configurations shows you how good your choice of ROM and Kernel are, and thats it.
So while it is not reliable to compare the scores of two different phone's but IMHO it is OK to compare results with people with the same smart phone.
I completely agree with you if you argue that a phone is a phone and is not a PC that you use for 3D stuff, but higher scores in Quadrant don't mean that I can play the latest 3D game and you don't, it's just that with higher scores is probably more likely that your phone is working more smoothly than others!?
And about the battery not lasting long when you get such high scores, I have to tell you my friends, if you want your battery last longer then what in gods name are you doing here, you should have probably not even upgraded your phone to Froyo, let alone GB or even the custom ROM that you are using now.
We all want to make our phones more interesting, put more functionalities in them and make them do more and more stuff every day, and with those desires we have to sacrifice something in return, and again IMHO a little bit of battery lifetime is the least I am prepared to sacrifice for the good looks and functionalities that my phone has compared to a stock just out of the box SGS.
Besides, I have a charger that I don't even use, there is a car charger in my car, a usb cable at home near the lap top and another one attached to the desktop and I have the same situation at work, so wherever I go there is the possibility to charge my phone, I really DO NOT understand it when people talk about battery lifetime.
Good Luck to All of YOU Modders & Testers & Adventurers
it is indeed fun when you finally arrive at the kernel that just fits your phone and needs the best. Be prepared for a whole series of "flames" from those who will variously dismiss quadrant and/or your choice of kernel because it's not their choice.
Ignore them: the rom and kernel choice is always a personal preference.
1.2ghz, bigmem and no journal ext4?? Thats the only way those scores are possible lol.
shomanadeh said:
Just wanted to share with you guys the Quadrant Advanced Scores of Latest version of TalonDEV Kernel.
It is amazing, I've used Semaphore, CF-Root, Dark Core, Galaxian and SpeedMOD before, but never before I have seen these astonishing results, so go ahead and install it and see for yourself.
Click to expand...
Click to collapse
Quadrant scores aren't that reliable. Froyo 2.2.1 has lower scores than Gingerbread 2.3.4 but some Froyo roms run smoother.
Anyway, it would be nice to replicate those scores? Which Talon kernel exactly are you using? Are you on a custom rom?
This is an interesting data if you base your choice on quadrant. But with all my respect to your post: High quadrant scores just means high quadrant scores. A phone it is not PC graphic card, somebody in the android community should develop a better way to compare performance for day-to-day users.
Shinny graphics but most important for a pocket phone is smoothness on everyday usage, stability and battery consumption. Really, the Quadrant or whatever score is the least concern most people should have on choosing a kernel, be the score thrusthworthy or not.
. .
shomanadeh said:
And about the battery not lasting long when you get such high scores, I have to tell you my friends, if you want your battery last longer then what in gods name are you doing here, you should have probably not even upgraded your phone to Froyo, let alone GB or even the custom ROM that you are using now.
We all want to make our phones more interesting, put more functionalities in them and make them do more and more stuff every day, and with those desires we have to sacrifice something in return, and again IMHO a little bit of battery lifetime is the least I am prepared to sacrifice for the good looks and functionalities that my phone has compared to a stock just out of the box SGS.
Besides, I have a charger that I don't even use, there is a car charger in my car, a usb cable at home near the lap top and another one attached to the desktop and I have the same situation at work, so wherever I go there is the possibility to charge my phone, I really DO NOT understand it when people talk about battery lifetime.
Click to expand...
Click to collapse
Well, I have about 3 days with CM7 stock on medium usage. I wouldn't sacrifice that, I get out a lot and for long so battery is of prime importance, as is mobility, independant of where or how many chargers I have.
To each his own I guess, but performance numbers come low last after stability, autonomy and performance (felt) for me.
On Semaphore 1.8 quadrant can be the same 3300-3444
Sent from my GT-I9000 using xda premium
Or platypus:
http://forum.xda-developers.com/showpost.php?p=15132880&postcount=1380
I got 3910 on damian gto.
Running at stock 1000mhz on jvt
Sent from my GT-I9000 using Tapatalk
Are this scores based on statistical methods?
Possible approach:
1. Stripping down the device to a level where a minimum of interference from running apps occur. A comparable level.
2. Collect minimum 5 scores.
3. Throw out top and bottom, calculate average based on the remaining values.
If not you're like comparing how fast you can move from A to B by changing engine in your car and forget the load.
Sent from my GT-I9000 using XDA App
It's Just for FUN!!! NOT Advancing any Science Here!?!
Are this scores based on statistical methods?
Possible approach:
1. Stripping down the device to a level where a minimum of interference from running apps occur. A comparable level.
2. Collect minimum 5 scores.
3. Throw out top and bottom, calculate average based on the remaining values.
Click to expand...
Click to collapse
AAAAAAA, No, these results are not based on statistical methodes!!! and NO, non of us and not even the 150000+ people that download these test apps have tested their phone in an labratory environment nor have they created an similar environment, these are all good old home baken test results, which I dont mind at all if I may say so myself.
It's just fun to test stuff and compare results, it's not that we are measuring **** sizes or something , and we are certainly not doing it for the good of man kind or advancing science, at least I hope not.
If not you're like comparing how fast you can move from A to B by changing engine in your car and forget the load.
Click to expand...
Click to collapse
But isn't this how they do it on the race track or when we street race? Try to strip the car down and make engine more powerfull?
Bench mark
i think that benchmarch did not represent the real feel of a kernel. Take a look to Midnigth Kernel, the quadrant score in not much but the phone is really smooth
---------- Post added at 01:25 PM ---------- Previous post was at 01:25 PM ----------
i think that benchmarch did not represent the real feel of a kernel. Take a look to Midnigth Kernel, the quadrant score in not much but the phone is really smooth
---------- Post added at 01:26 PM ---------- Previous post was at 01:25 PM ----------
O-T said:
Are this scores based on statistical methods?
Possible approach:
1. Stripping down the device to a level where a minimum of interference from running apps occur. A comparable level.
2. Collect minimum 5 scores.
3. Throw out top and bottom, calculate average based on the remaining values.
If not you're like comparing how fast you can move from A to B by changing engine in your car and forget the load.
Sent from my GT-I9000 using XDA App
Click to expand...
Click to collapse
i think that benchmarch did not represent the real feel of a kernel. Take a look to Midnigth Kernel, the quadrant score in not much but the phone is really smooth
hi. i have seen people here saying that "Benchmarks prove nothing" and even one of the Senior member saying so.
but what i have experienced after using Note and switching between different ROMS like Rocket Rom v22 and v23, Midnote, CheckRom and Stunner Rom, i have found that Yes the Benchmarks Do matter a lot in terms of just explaining how your phone will perform.
i mean i ran the bencmarks on all of these Roms and i used them all for about 4 to 5 days each, and i found that not all of these give equal results and not all of these are equal.
the Check rom performed the worst among all of these Roms. by the time i installed it, my phone started giving me issues. like the battery issue. whenever i connected the phone to the data cable and either connect to my pc or laptop or even the charger, there would a screen popup saying, battery removed! i did everything i could do to fix this issue but all in vain.
well the best Rom in terms of performance is the Rocket Rom. rocket rom gave me the highest scores and i found it to be the most stable Rom.
Midnote was ok rom. it gave me really good benchmark scores but there were lots of force closes.
i really dont want to talk about the CheckRom. it gave me the lowest scores and it proved to be the most buggy rom i have used, and then came the battery removed issue.
Surprisingly i, the ICS Stunner Rom was the Rom which gave me around 110000 score in the Browser mark in the native browser, but for the rest of the benchmarks it gave me normal scores.
so in my point of view the best rom for note for now is the Rocket rom, but if u want buttery smooth UI and 199% smoothness, go for the ICS Stunner. but the payoff is that u loose the SNote and the screen capture capability which i really love about my note.
hence proved, Benchmark Results Do make a difference!...... These are my own personal results and you may have different results but I thought I would share my findings....
aami.aami said:
hence proved, Benchmark Results Do make a difference!
Click to expand...
Click to collapse
Hardly proved! Far from it.
Benchmarks *can* give you an indication of performance, but *nothing* is better than simply trying out different ROMs and making your own objective decision.
Fwiw though, some benchmarks simply aren't worth the time and effort - Quadrant, I'm looking at you!
Regards,
Dave
Sent from my GT-N7000 using Tapatalk 2
I am gonna steal my friend's Chasmodo quote (credits to Chas)
A man without a benchmark is like a camel without electric toothbrush
end
Sent from my GT-N7000 using Tapatalk
Benchmarking could be slightly usefull when comparing different roms though optimalizations that are specifically made to reduce boottime, remove lag or whatever might not show up in te scores.
Using benchmarks to compare phones is definately useless, for example because Android forces you to run apps in the native phone resolution. The Note runs at 1280x800 which means any phone with a similar GPU but lower resolution like the standard 800x480 could synthetically outperform the Note.
Just to throw another point in there I know it can be easily disputed but take for example iOS and windows phone.
Windows phones are limited to single core and they still run smoothly due to optimization.
the latest iPhone 4s has a dual core A5s @ 800mhz, yet it still runs smoothly.
As mentioned above in the thread the only real way is to try out the phone for yourself.
Dude, it's obvious you're no scientist. You haven't proved anything trying to connect benchmark scores with stability and force closes. The only thing a benchmark could indicate is possibly smoothness. It's all about you simply trying different ROMs and finding one that is subjectively working best for you.
Sent from my superior GT-N7000 using Tapatalk
Benchmarks generally test specific capabilities of hardware: arithmetic calculations, GPU performance, etc.
That being the case they are a good yard stick for how the HARDWARE of different devices compare in terms of performance for the most part. Although software and the OS do obviously play their part, the tests are generally targeted at exercising specific aspects of the 'computer' rather than the overall OS.
OS performance is another, much more complex, aspect that benchmarks struggle to cater for. Small changes in benchmark scores between different ROMs is unlikely to tell us much. Somebody who reviews those ROMs and give a fair summary of the overall performance, stability, etc is a much better guide (like the OP, in this case - after a fashion!)
Check Rom is '****ty'?
What utter claptrap. Only proves you have no idea about how to set your phone up properly.
As for the battery issue, really? I've been running checkrom on 3 notes(mine,my wifes and my dads) and never had fcs, battery issues.
Benchmarks are for people like you.
aami.aami said:
the Check rom is the Most ****ty Rom i have used among all of these Roms. by the time i installed it, my phone started giving me ****ty issues.
Click to expand...
Click to collapse
Yo, Benchmarker, watch your language.
Calling any development '****ty' just showcases your poor manners, egomania and general absence of intelligent thinking.
The devs give you something for free.
You don't like it, move on. And keep shtum about it. That's what people with manners do.
I wish you luck in your future benchmarking endeavors, and hope never to hear about them.
Benchmarks don't prove a ****. But I do it every time I try out a new kernel and waste 5 precious minutes of my life.
By the way, calling Checkrom ****ty was insult of all checkrom devs and ultimately an insult of xda itself, we are here because of all these devs.
You simply could have said it had lowest benchmarks or even it was sub par.
May I hope that next time you will carry general code of conduct and keep your ****ty opinions within your skull ?
chasmodo said:
Yo, Benchmarker, watch your language.
Calling any development '****ty' just showcases your poor manners, egomania and general absence of intelligent thinking.
The devs give you something for free.
You don't like it, move on. And keep shtum about it. That's what people with manners do.
I wish you luck in your future benchmarking endeavors, and hope never to hear about them.
Click to expand...
Click to collapse
That's what I'm talking about.
Don't benchmark the OS and mobile, but benchmark your brain that is it capable enough to run that device at its fullest without the need of any score to prove anything.
Just my 2 cents.
aami.aami said:
hence proved, Benchmark Results Do make a difference!
Click to expand...
Click to collapse
where have you proved anything? you really are a bit deluded arent you
Benchmark are valuable in terms of relative comparison to the same benchmark.
Benchmark just don't correlate well to real world performance.
IE: Note LTE ATT on speed app runs 15-20 M's, mine runs 7-9 in same location. Same network. Yet download Engadget simultaneously and they're even.
Benchmarks prove nothing.
/wisdom
mfractal said:
Benchmarks prove nothing.
/wisdom
Click to expand...
Click to collapse
Benchmarks prove everything.
/wisdumb
Zamboney said:
Dude, it's obvious you're no scientist. You haven't proved anything trying to connect benchmark scores with stability and force closes. The only thing a benchmark could indicate is possibly smoothness. It's all about you simply trying different ROMs and finding one that is subjectively working best for you.
Sent from my superior GT-N7000 using Tapatalk
Click to expand...
Click to collapse
i really dont understand why some people dont understand what the main point some statement is. its quite simple, nothing scientific and i did not talk about any technicalities. see for your self, the more smoother the ROM is the more better the Hardware of the phone is going with the build of the ROM that's obvious, right? now that would mean more room for the hardware of the phone to be available for better benchmark results.
aami.aami said:
i really dont understand why some people dont understand what the main point some statement is. its quite simple, nothing scientific and i did not talk about any technicalities. see for your self, the more smoother the ROM is the more better the Hardware of the phone is going with the build of the ROM that's obvious, right? now that would mean more room for the hardware of the phone to be available for better benchmark results.
Click to expand...
Click to collapse
probably but not necessarily, correlation does not mean causation,
each benchmark result is moreless the weighted average of few or more individual test results - now let's take Quadrant, which is (was?) heavily biased towards 3D graphics performance - how much is that representative of ROM performance under typical (non-gaming) phone usage?
PS.
Benchmarks may matter. Size matters more.
(Anyone telling otherwise is either ashamed or trying to be nice)
From the Moderator
Watch your language....... nothing wrong with arguing about results or subjective findings, but almost all opinions are subjective..... so discuss nicely or we close the thread if flaming starts up
Thanks.... oka1
One thing apple got google and its oem beats. Dont fking look at the hardware and benchmark. Look at the user experience. Not saying ios and iphone is great for the folks who like to tinker. But for a dual core 800mhz iphone4 or ipad2 it sure runs a lot smoother than a note with stock touchwiz or some other launcher. Afterall u r using ur device, ur own judgement should tell u how good the device is not some program score. And there will always room for hardware improvement, but can the software follows? How many app is actually utilizing multi core process?
Sent from my GT-N7000 using Tapatalk 2
put new Kernels you will see big different in benchmark.
From my experience testing roms + kernels I see that benchmarks really nothing
I got with Antutu benchamrk with stock rom and stock kernel about 6900 score
but it was so laggy and battery sucks. atm using Chrack´s rom extra small + speedmod v10 and its perfect no lags good battery life smooth but the benchmark sucked got like 4500. So benchmarks aren't the way to chose a rom otherwise there is no need to test and share experience with people