Related
Results in Google Docs Spreadsheet
All ICS ROMs Benchmarked
Any regular visitors to this thread can't have failed to notice the big changes in the spreadsheet. In particular I've been studying the effect of the kernel the ROM developer includes. If you look at column H, you will see whether I recommend flashing another kernel to improve performance at stock speeds.
REMEMBER: Performance is not everything! Many custom kernels provide extra features!
If you do decide to switch kernels, and only desire stock speeds and a reduced feature set, I recommend two kernels in this post.
If you are new to this stuff read this.
SUMMARY OF RESULTS
#1st place: Pixel ROM 1.72
#2nd place: CyberGR-MOD|NS.NGN ICS v10 HYBRID
#3rd place: MIUI Catalyst 2.4.6
#4th place: AOSP+ N.5
#5th place: [MIUI][2.4.6] Oodie pRiMe rOcket fUeled
Where do I find all these wonderful ROMs?
Their XDA threads are all linked in the table, so click through to discover each ROM's feature set, included apps, theme, and evidence of good karma!
Looking for a Gingerbread ROM? Try this.
Where did all your other benchmarks go?
Kernel Features & Benchmarks: this thread
Battery drain: this thread
CPU Governors and I/O Schedulers: this thread
Power Saving Governors: this thread
Thanks to all the developers.
Testing Methodology
Each ROM is clean installed, allowed its own default settings, then all benchmark software is installed and run once the system is settled. I also perform one screen off screen on, just out of habbit (this used to resolve a graphics bug in gingerbread, which I'm not even sure exists in ICS). All results are recorded in the spreadsheet. Then I restart in recovery, wipe cache and dalvic (thanks morfic for dalvic wiper) and install morfic's T132-I kernel. Reboot, use nstools to select performance (default I/O scheduler for morfic's kernel is deadline). Reperform all benchmarks, recording them in the spreadsheet. The calculation for the overall CPU RAM I/O and GRAPHICS scores is identical to the one I use in my kernel benchmarking spreadsheet. Please refer to that thread for more detail. However, there is one difference. Quadrant behaves very differently in different ROMs depending on what graphics tweaks they incorporate. As such, no Quadrant scores are used in this ROM benchmarking study. For more details, see under 'Graphics' below.
The final step was to average out the results of both kernel scores, and ranking on that basis. Some would say that they are best ranked by looking at T132 scores only. I would say that is a fair point, and definitely worth consideration.
There are two reasons I incorporate the default kernel score:
1) Overall scores have a worst case scenario accuracy of within 2% of the true mean due to variability in the benchmarks (more detail on how I obtain this figure is available in my kernel benchmarking thread). For normally distributed data (which we can safely assume these benchmarks produce) averaging two sets of results will result in doubling the accuracy to within 1% of the true mean, making the ranking more reliable.
2) Combining the default kernel Scores for ROMs that used a poor performing kernel, or selected bad governor/scheduler combinations by default will suffer as a consequence. These problems can be solved if the user flashes their own preferred kernel and/or adjusts the kernel settings. However, many new users will not perform this step. For that reason, I want to give a score that at least somewhat reflects the performance as the developer intended.
-Graphics
One of the early findings is selecting force GPU rendering in the developer options improves 2D performance, in Quadrant only. Also, and only in Quadrant, 3D performance can be improved by deleting or moving /system/lib/egl/libGLES_android.so using root explorer or equivalent. These tweaks/hacks do not seem to affect other benchmarks in the slightest.
Because of the bizarre effects of these tweaks on Quadrant scores, I have removed them from the formula that calculates the overall GFX score, and this in turn impacts the overall average score. In short, it makes comparing each ROM's score fairer.
You can see evidence of how the graphics tweaks make Quadrant behave differently at the bottom of the spreadsheet, highlighted in light red. The host ROM for this particular comparison was DianXin or DX ROM for short. This is a reference to a post I make in DX ROMs thread, where I first decide that removing Quadrant is the best option: http://forum.xda-developers.com/showpost.php?p=22323688&postcount=194
-Android 4.0.1
These ROMs (at the bottom of the spreadsheet, highlighted in light red) were based on early maguro ports provided by koush. I have left them in, because although they are difficult to compare to 4.0.3 builds with the correct drivers and whatnot, they do compare to each other in one important respect: the kernel. One is stock, the other is built by eugene373 (galaxy nexus/nexus s dev.) Using eugene373's kernel, CPU, I/O, and Graphics were all raised to a high standard that is on a par, or above some of the current ROMs. This goes to show that a customised kernel can offer great benefits, even on a ported ROM.
-Android 4.0.4
As of the 5th of February, my table contains all the ICS ROMs that are listed in the XDA forums. There was only one 4.0.4 version at the time, and this was the best performing in the benchmarks. When the source code is released, I'm sure many ROMs will update. At this point they will benefit from increased RAM scores and javascript performance. If all the ROMs scores improve by the same proportion, the rankings will not be affected. I will attempt to included major releases, but I will not reperform these tests every time there is a new nightly.
Reserved 4
Reserved 3
Thanks
I was waiting for this
It might lead to troll wars though
By the way, i dont see Pete ICS rom in there, im using it.
Ibn Saeed said:
Thanks
I was waiting for this
It might lead to troll wars though
By the way, i dont see Pete ICS rom in there, im using it.
Click to expand...
Click to collapse
Thanks, and no troll wars please y'all.
Don't you think "popularity" is going a little too far? It will definitely lead to fights...all the devs are good and I don't think prominence is a good indicator of a good rom
But it's up to you
Sent from my Nexus S using XDA App
Personally, benchmarks are very misleading... Some of the most stable and well developed roms don't get the greatest scores because they weren't made with benchmarks in mind.
Sent from my Nexus S 4G using xda premium
Another great work! Thanks
Sent from my Nexus S using XDA App
mathkid95 said:
Don't you think "popularity" is going a little too far? It will definitely lead to fights...all the devs are good and I don't think prominence is a good indicator of a good rom
But it's up to you
Sent from my Nexus S using XDA App
Click to expand...
Click to collapse
just remember, popular doesnt mean better. but to get angry at the truth(whatever it might be) is just silly. its lies that you should be upset with.
Where's Peter Alfonso's ROM? Bugless Beast is a mainstream rom, second only to Cyanogenmod in scope and breadth.. The ROM thread is in the General forum, because he links to his page at Rootzwiki, but it's a more prominent ROM than many you've listed.
Please include it in the benchmarking as well.
Sent from my Nexus S using Tapatalk
http://forum.xda-developers.com/showthread.php?t=1430170
you didn't add and test this one
please add pete ics bugless beast.
Omissions fixed!
UPDATE: kwiboo's project removed, was an SDK port. So so tired now, got to sleep. Tomorrow begin again with Brainmaster's tweaked ICS
Which kernel is used to make the tests?
-----
Forget It...
"I'll be benchmarking them twice with whatever kernel they come with, and then with morfic's bare boned T132"
biotecsoul said:
Which kernel is used to make the tests?
Click to expand...
Click to collapse
Read the op.
Sent from my Nexus S 4G using xda premium
u could also add overnite battery drainage column with sync on or somethin like that, if not overnite then a period of 2 hrs idle .. how much does it drain ..
Holy **** man, what would it be if the NS comunity didnt have you??
Thumbs up!!!!
Sent from my Nexus S using xda premium
thegtfusion said:
Holy **** man, what would it be if the NS comunity didnt have you??
Thumbs up!!!!
Sent from my Nexus S using xda premium
Click to expand...
Click to collapse
Haha, that makes me feel good. Wait, Benchmarks > XDA thread > Positive XDA user feedback > Dopamine release = Addiction to benchmarking? Time to join crackflashers anonymous...
mathkid95 said:
Don't you think "popularity" is going a little too far? It will definitely lead to fights...all the devs are good and I don't think prominence is a good indicator of a good rom
But it's up to you
Sent from my Nexus S using XDA App
Click to expand...
Click to collapse
I doubt it would lead to fights between devs, well hopefully not. I can see users arguing stuff more than devs.
Benchmarks aren't that good anways.
BTW, your kernel sucks!!!!
Just kidding been running smoothly since I started using it the other day
-----------
Thanks for the ratings system. Looks nice!
So, the title says it all.
I'm curious about what governor do all you guys use??
Place your vote here.
Also, feel free to comment why you choose your governor!
se1988 said:
So, the title says it all.
I'm curious about what governor do all you guys use??
Place your vote here.
Also, feel free to comment why you choose your governor!
Click to expand...
Click to collapse
Lagfree, on Marc's Kernel. Awesome, but not in your poll.....
I think this should be moved to q&a.
My favorite: smartassv2
Why? It includes a stanby management, is stable and fast.
Sent from my HD2 using XDA Premium App
se1988 said:
So, the title says it all.
I'm curious about what governor do all you guys use??
Place your vote here.
Also, feel free to comment why you choose your governor!
Click to expand...
Click to collapse
You shoulda posted in Q&A
But i use smartassv2 with my OWN custom settings and i use it because it gets great performance with amazing battery life
I am still working between different settings on TBD. Lagfree with Marc's kernel is good but I am now working with smartassv2 with custom settings. So far so good
Lagfree 128 - 1190
btw, this should be in the Q&A
It sucks that majority of governors have no description, so only way to decide between them is through trial and error (which is very tedious given the number of governors and differences may only be placebo) or by reading other's recommendations.
I was using convervative with a heavily customized settings, then smartass/V2/smoothass (switching between them, AFAIK they are very similar anyway), but now I'm testing Hyper as it's default on new DORIMANX.
moved to Q&A
Not sure if this is a thread to post this sort of problems...
Anyway, at the moment I got a HD2 running flawlessly HyperDroid-CM7-v5.9.0 with smartassV2 governor and I'm very happy with it.
The problem is that as soon as I switch to for example interactiveX governor I start getting freezes where the phone is left screen on and nothing else working.
I've been previously dual booting wp7 and android sd. Back then I had same sorts of freezes on android but would just tweak my auto memory killer to less aggressive settings to not get freezes. I'm a bit confused on this but after all quite sure that this is about something else than memory related problem.
Any ideas?
I have also tried few of these governors, but I think that there is no big difference between the better ones like hyper, ondemand etc. just a placebo when somebody release "new" governor
Over my time here in XDA I've built up a few studies on the Nexus S.
Here are the links. There are summaries in each thread in the first and second (and sometimes third) posts of the main findings, but I've done some very quick and minimal overviews here also.
How to build a kernel: this thread
-This post takes you through installing Ubuntu right through to managing git, in order to compile your own kernel, and flash it in CWM recovery!
-Includes the tools you need to add MTP capabilities to your kernel.
User eXperience Benchmarks: this thread
-This shows which ROM and kernel threads have the best user feedback. The feedback is processed by an external sentiment analysis API. It is only meant as a rough guide, something of a quick reference particularly if you are looking for a new ROM to try. As of June 2012 this is no longer maintained.
All ICS ROM Benchmarks: this thread
-Freely available benchmark programs were used to determine which ROMs had the best performance. Generally the difference between the best ones is very small.
Battery Drain Benchmarks: this thread
-A collection of tips for getting the best battery life, and dispelling some myths.
Kernel Features and Benchmarks: this thread
-My original study, the one that started it all, the kernel benchmarks! Provides a quick table to compare which kernels perform best in CPU/RAM/IO/Database/Graphics.
CPU Governors and I/O Schedulers: this thread
-A study done to find which schedulers perform best in combination with the most popular governors. My two favourites (ondemand and smartassV2) combine best with noop.
Power Saving Governors: this thread
-A meta study done combining the findings from the battery study and governor study to show which governors will save the most power. Ondemand comes out significantly better than conservative, which surprised me. SmartassV2 is the best compromise for high performance and efficiency.
Does SuperCharging work?: this thread
-This was just a short study to find out if this script is suitable for the Nexus S, but it turned out it had no effect, positive or negative.
Kernel Memory Allocators: this thread
-This is another short study for kernel developers to show that SLUB performs best.
====
This thread is just an index. Our new mod veyka agreed would be a good idea to summarise all my studies for addition into the sticky roundup.
====
EDIT: Feel free to suggest any new things that I might be able to test!
im out of thanks for today.. but, awesome work regardless. thank you
simms22 said:
im out of thanks for today.. but, awesome work regardless. thank you
Click to expand...
Click to collapse
Me too, but thank you too!
Edit: ah just use the app, unlimited thanks!
bedalus said:
Me too, but thank you too!
Edit: ah just use the app, unlimited thanks!
Click to expand...
Click to collapse
really? unlimited? i dont use the app though, just the browser.
Uh, well, I haven't tested the limit... but I never get denied!
All your work was great I battery drain thread was very helpful
Sent from my Nexus S
Thanks for all your hard work.
IF , one of my assassins had done their job ..........
.. Generated via Tapatalk ..
Oodie said:
IF , one of my assassins had done their job ..........
.. Generated via Tapatalk ..
Click to expand...
Click to collapse
When they finally come for me, I will benchmark their performance.
bedalus said:
When they finally come for me, I will benchmark their performance.
Click to expand...
Click to collapse
nice answer
And nice guide for your very instructive benchmarks bedalus
Even when browsing through the app you'd still get that annoying thanks limit.
This is a really good overview for new NS users. (I doubt there are ever some since mostly would go directly for G-Nex)
What's next?
That "Oodie's Assassins' Performance Benchmark" sounds interesting, though.
PS: @Oodie, you should have kanged your assassins before sending them to bedalus' compound. That way, your assassins' would have at least injured bedalus
UPDATE: Memory Allocators studied. SLUB wins!
Great reference for a refresher or for a new comer. Really like the battery drain benchmarks and how you discuss some myths. Governors is great...love Smartass V2 with Matrix V20!!
Thanks @bedalus!! Totally deserves a sticky.
Sent from my Nexus S using xda premium
Thanks. It's in the sticky roundup!
bedalus said:
Thanks. It's in the sticky roundup!
Click to expand...
Click to collapse
Can we vote for the sticky or it is just for the moderators?
There was like ten stickies, I think veyka wants to keep everything in a list in one sticky.
I appreciate the sentiment though!
bedalus said:
There was like ten stickies, I think veyka wants to keep everything in a list in one sticky.
I appreciate the sentiment though!
Click to expand...
Click to collapse
thank you for all your work, love your kernels ... currently been using deadline/ondemand but since you say it is not best I will try smartassv2+noop.. (I've found smartass with other combinations give me a lot of lag)
BLaZeDRas said:
thank you for all your work, love your kernels ... currently been using deadline/ondemand but since you say it is not best I will try smartassv2+noop.. (I've found smartass with other combinations give me a lot of lag)
Click to expand...
Click to collapse
Ondemand for good balance between performance and battery saving. Smartassv2 for gaming. CFQ and deadline are the best governors, but deadline probably has the edge.
Thanks! Now this widens my perspective on choosing my own Governor and Scheduler Combo.
It seems that with the Devil kernel, I've had a few more options on choices. After reading a few posts and other items - including a plethora of bad/wrong advice, things seem to be running best the the Lulzactive CPU Governor and the IO Scheduler set to BFQ. When I say best, I mean that the phone has a balanced level of responsiveness even when a bunch of crap is running. It didn't seem to get totally sluggish and appear to be hung as seemed to be the case when the defaults were used previously.
I invite others to share their thoughts. Note - I don't care about battery usage within reason.
Smartass2 with zen has been my combo for a while now. Aokp jb m1 w/ devil 1.9.5. Experimented with a few other combos till I found this one and have stuck with it.
I'll give yours a try and see how I likes it
Sent from my SCH-I500 using xda premium
Looked at more info on SmartassV2 and it appears that depending upon settings that it may be faster than lulz but also a bit hungrier battery wise ...I think that one is gonna be one of those YMMV sort of things. As for BFQ, it is the one to choose when dealing with heavy i/o which somehow I seem to have on my phone...haven't figured out the why part, but it seems to keep on truckin' when put under load. I have not had luck in finding anything substantial showing what/how the zen io scheduler is configured. If you have any links, post away.
Best and Most Complete Summary Article of both CPU Governors and IO Schedulers that I've Found:
http://tinzdroid.blogspot.com/2012/07/android-kernel-governors-modules-io.html
I've pretty much always had Smartassv2 and sio just because it's what I read was the best combo (or at least most common). I realize Lulz is pretty good but from what I was reading it needs to be configured properly before you get good performance out of it, and quite frankly I don't want to deal with that. My performance overall has been excellent and it feels pretty snappy. Battery life... well it's pretty good, I can go a full day of fairly heavy usage without needing to run to a charger, so that's good enough for me. I should also mention that I undervolt my phone pretty substantially, so that definitely makes a difference in battery life, especially when the phone is in standby.
My source for governors comes from http://forum.xda-developers.com/showthread.php?t=1736168 and my source for schedulers comes from http://forum.xda-developers.com/showthread.php?t=1420742
Gave lulz/bfq a run for a little over a day and my report is that it isn't as snappy as smartassv2/zen. Just a little choppy scrolling up and down and left to right.
I used to use sio with smatrassv2, but I saw that zen was new to 1.9.5 and tried it - silky smooth on my mez.
Sent from my SCH-I500 using xda premium
mezster said:
Gave lulz/bfq a run for a little over a day and my report is that it isn't as snappy as smartassv2/zen. Just a little choppy scrolling up and down and left to right.
I used to use sio with smatrassv2, but I saw that zen was new to 1.9.5 and tried it - silky smooth on my mez.
Sent from my SCH-I500 using xda premium
Click to expand...
Click to collapse
yeah, I've tried lulz, but it didn't seem to do need any good so I switched back to smartassv2 and vr, but then I found out devil 195 had Zen and I wanted to see what it was all about and honestly it's been running quite amazingly with smartassv2
I've been rocking smartassv2 and sio best combo I can find
Sent from my SCH-I500 using Tapatalk 2
Hey everyone, I have been on the hunt for the last few days for the best Kernel for my N7, in regards to performance. Earlier today someone turned me onto M-Kernel. I must say, it is very impressive. It has a built in 1.9 GHz overclock, that I was able to set via no-frills. I am running Clean ROM.
This is a multi-purpose post. First off, I wanted to let everyone know about my experience because I had a little trouble figuring out the best combo. The second reason is I would like to know if anyone knows of any other ROM/Kernel combos to get the highest quadrant score possible.
I have a co-worker who has a Galaxy SIII, and we have a friendly rivalry. I would really like to blow his phone away with my Nexus.
Thanks!
Check the link in my sig for my guide to the 'best' kernels. I don't think there is a lot of difference between roms in terms of benchmarks results.
Sent from my Nexus 7 using xda premium
bcvictory said:
Check the link in my sig for my guide to the 'best' kernels. I don't think there is a lot of difference between roms in terms of benchmarks results.
Sent from my Nexus 7 using xda premium
Click to expand...
Click to collapse
Clemsyn's Elite Kernel CPU overclock to 1.8GHz and GPU overclocked to 700MHz
droidhaxxor said:
Hey everyone, I have been on the hunt for the last few days for the best Kernel for my N7, in regards to performance. Earlier today someone turned me onto M-Kernel. I must say, it is very impressive. It has a built in 1.9 GHz overclock, that I was able to set via no-frills. I am running Clean ROM.
This is a multi-purpose post. First off, I wanted to let everyone know about my experience because I had a little trouble figuring out the best combo. The second reason is I would like to know if anyone knows of any other ROM/Kernel combos to get the highest quadrant score possible.
I have a co-worker who has a Galaxy SIII, and we have a friendly rivalry. I would really like to blow his phone away with my Nexus.
Thanks!
Click to expand...
Click to collapse
http://forum.xda-developers.com/showthread.php?t=1889571
Motley is an overall great kernel