New to this stuff? Read this.
Features: plain text or FEATURE TABLE
All ICS Kernel BENCHMARKS
Where did the other benchmarks go?
All ICS ROM Benchmarks: this thread
Battery drain: this thread
CPU Governors and I/O Schedulers: this thread
Power Saving Governors: this thread
Thanks to all the developers.
This information regards the Kernel Benchmark spreadsheet.
Sheet 1: The Overall Scores
I realised that looking at all the raw data was going to give people a headache. (For those who enjoy headaches, like me, the raw data is still visible lower down.)
How did I arrive at the percentages?
First, I looked at an individual result, e.g. the AnTuTu 2D graphics score for simplekernel. I use it as the numerator of a fraction where the denominator is the average of all the AnTuTu 2D results from all the kernels. If above 1, this score is better than the average. If less than 1, this score is below. I do the same thing for all the graphics results, and then calculate the mean. This provides equal weighting for all the various graphics benchmark results. In other words, it prevents a bad quadrant result from skewing the overall picture if the rest of the graphic results are good.
How to interpret these percentages? At the time of writing, if we compare morfic's RAM result of 109% to the terryhau's stock(ish) simplekernel RAM result of 96%, we can say with some certainty that morfic's RAM performs 13% better than stock.
Sheet 2: Statistics for Variance
Firstly, thanks are due to morfic and ezekeel whose scepticism of benchmarks made me want to investigate the issue of variance in the results.
The most important column is n. Check out these rows:
Row 8: AnTuTu's Database I/O benchmark can vary by as much as 7.55%
Row 16: Quadrant Advanced Memory benchmark can vary by as much as 6.66%
Row 21: Smartbench 2012's Gaming Index can vary by a massive 24.42%
Row 25: Vellamo's Aquarium Canvas can vary by 12.11%
Row 26: Vellamo's Pixel Blender can vary by 5.66%
Row 28: Vellamo's Aquarium Canvas can vary by 5.59%
These figures are calculated within two standard deviations of the mean, or in other words, a 95% certainty. For example, we can say that 95% of the time, the Quadrant Advanced Memory benchmarks we run will give a result within 6.66% of a genuinely accurate figure. This is okay, but just not good enough for me!
By eliminating these test results, I have improved the accuracy of my overall scores to within 2% of the genuine mean, with a 95% certainty. In the spreadsheet, you can see the individual accuracies for each of the overall scores: CPU, RAM, I/O, GRAPHICS. Red highlighting of the figures takes these individual variances into account, and shows which tests are definitely at least 1% better than stock, even assuming stock tested at it's maximum, and the compared kernel tested at it's minimum.
Testing Methodology
To test the kernels I want a fair environment, so any differences in the results are down to the kernel, and nothing else. To achieve this I first make sure I have a clean system:
1) Format /system and /cache and wipe dalvic
2) Install the ROM, install the kernel
3) Boot up, use nstools to select deadline for I/O scheduler
4) Then select ondemand governor and set the minimum frequency to equal the maximum, which is not the same as performance, but close enough if you see my CPU governor comparison chart. CPU cycles are wasted as the governor tries to decide what frequency to jump to, even if it has no option! Thanks to Steve Garon for the info. This method was suggested by morfic, thanks again! I/O scheduler is set to deadline. Thanks tchaari!
5) Restore all my benchmark programs. Set everything to off, no gps, no sync, no BT, airplane mode. Force GPU rendering is selected. Wifi is on for connecting to the benchmark servers. A power adaptor is in use so the battery is always full.
Then I begin testing:
6) Power off and power back up. When lock screen arrives, wait one minute to settle the system, i.e. until screen auto-off. Count to three. Unlock, and begin testing, recording all scores at the bottom of my spreadsheet. I copy them up (from the old data to the current raw data section) once complete. This avoids the overall score constantly altering for any current viewers, and means I keep a record of all previous tests under the heading Old Data.
Source Threads For Kernels and Stated Features
Feature Comparison Table: https://docs.google.com/spreadsheet/ccc?key=0AntDDKv-lS6IdHlOVUZ1WkJtMmpXdjl6TURCVFR0cWc
r_data (Air Kernel): http://forum.xda-developers.com/showthread.php?t=1469842
- I/O Schedulers: cfq, deadline, noop, sio (Default is cfq)
- CPU governors: conservative, ondemand, performance, lazy, lulzactive(v2), smartass, smartassV2, intellidemand, interactiveX, scary, lagfree Lionheart (Default is ondemand)
- BLN, BLD, Touch Wake and custom voltage by Ezekeel
- Overclockable to 1.2Ghz
- SQLB
- BIGMEM
- init.d support
- Voodoo sound and colour
- Deep Idle
- Live OC
- CPUIDLE
- 65 fps
- Blue Colors
- BIGMEM
- init.d support
- USB OTG support
- Great Battery Life!
- CIFS, TUN support
- SLQB memory allocator
- Optimized CRC32 algorithm
- ARM RWSEM algorithm
netarchy: http://forum.xda-developers.com/showthread.php?t=1413117
- I/O Schedulers: cfq, deadline, noop (Default is deadline)
- CPU governors: interactive, conservative, userspace, powersave ondemand, performance (Default is ondemand)
- Overclocking up to 1.4ghz enabled
- Voodoo Sound driver
- Voodoo Colour
- CIFS support
- TUN support
- NFS support
- Updated kernel base (Currently 3.0.14)
- Cpu policy fix to avoid crashes with SetCPU and screen off profiles
- EXT4 remount tweak*
- BLN
- Custom Voltage Interface
- Deep Idle
- Backlight Dimmer
mathkid95: http://forum.xda-developers.com/showthread.php?t=1361477
- I/O Schedulers: cfq, bfq, deadline, noop, SIO, V(R) (Default is deadline)
- CPU governors: lulzactive, interactiveX, minmax, smartassV2, lazy, ondemand, userspace, performance, intellidemand, lionheart, wheatley (Default is ondemand)
- Based on Linux kernel version 3.0.14
- OC to 1.46 Ghz
- Voodoo sound and colour
- BLN
- BFS
- SLQB memory allocator
- Stochastic Fair Blue (internet packet queue management algorithm)
- CIFS, TUN, NFS
- INIT.D*
- Deep Idle
- LiveOC
- BLX
- BLD
- Ezkeel's voltage mod
- Touch Wake
- Bigmem
Ezekeel: http://forum.xda-developers.com/showthread.php?t=1336529
- I/O Schedulers: deadline, noop (Default is deadline)
- CPU governors: interactive, conservative, ondemand, userspace, powersave, lazy, performance (Default is ondemand)
- Based on stock Android kernel 3.0.8 IML74K
- CPUfreq governor 'lazy'
- Live OC version 1
- Custom Voltage version 1
- Battery Life eXtender (BLX) version 1
- Power saving state DEEP IDLE version 2
- Touch Wake version 1
- Backlight Dimmer (BLD) version 3
- Backlight Notification (BLN) version 9
- Seperate BFS and CFS versions available
- SLQB memory allocator
- Voodoo Color version 2
- Voodoo Sound version 10
- CIFS
- TUN
- Bigmem (+20 MB more available RAM)
Steve.Garon: http://forum.xda-developers.com/showthread.php?t=1427352
- I/O Schedulers: cfq, V(R), deadline, noop, sio (Default is cfq)
- CPU governors: lulzactive(v2), smartassV2, lazy, ondemand, conservative, performance (Default is conservative)
- Boots @ 1.1Ghz Overclock (Bus 220MHZ)
- Undervolted to maximize battery life
- Voodoo Sound V10
- BIGMEM for more available memory
- Kernel updated to 3.0.16
- Increased refresh rate 65fps (alternate version 56fps/improved colour available)
- Backlight Notification
- Backlight Dimmer
- Battery Life Extender
- TouchWake
- Custom voltage
- CIFS Module built-in
- Cold colors
- Cpuidle backport from 3.2 Kernel
- up to 1320 Mhz overclock
- init.d support
Morfic: http://forum.xda-developers.com/showthread.php?t=1036996
- I/O Schedulers: deadline, noop (Default is deadline)
- CPU governors: conservative, ondemand, userspace, powersave, interactive, performance (Default is ondemand)
- BAREBONES i.e. no features
- TEUV version has BLN and BLD
Terryhau: http://forum.xda-developers.com/showthread.php?t=1420542
- I/O Schedulers: cfq, deadline, noop (Default is cfq)
- CPU governors: conservative, ondemand, userspace, powersave, interactive, performace (Default is ondemand)
- BLN
- BLD
- CIFS module
- Init.d
- Insecure (ro.secure=0)
- Voodoo (*optional)
Reddv1: http://forum.xda-developers.com/showthread.php?t=1173797
- I/O Schedulers: conservative, ondemand, userspace, powersave, interactive, performance (Default is ondemand)
- CPU governors: cfq, deadline, noop (Default is cfq)
- UV
- Voodoo Sound V.10
- init.d Support
- ext4 optimizations
- BLN
- Compatible w/ NS & NS4G
- BLD
- Touch Wake
- Deep Idle
- Live OC
- Custom Voltage
- ICS version, ADB/ UMS/ Wifi tethering/ USB tethering/ Data stats & limits all work
- ICS version, ext4 optimization, init.d support and all mods from GB version
- Lazy CPU governor (but not in testing version)
Eugene373: http://forum.xda-developers.com/showthread.php?t=1401455
- I/O Schedulers: cfq, bfq, deadline, sio, noop, V(R) (Default is cfq)
- CPU governors: minmax, lazy, lulzactive, lagfree, interactive, interactiveX, conservative, ondemand, userspace, powersave, smartass, smartassv2, scary, performance, brazilianwax, intellidemand, lionheart, ondemandx, savagedzen, Wheatley (Default is ondemand)
- OTA ICS Compatible
- ro.secure=0 ( So you can push file's via adb )
- Voodoo Sound
- Voodoo Color
- Added BLD
- Touch-wake
- BT Sleep
- LCD voltage drop
- Bigmem patch
- BLN
- TUN
- CIFS
- Battery 100%
- added dm-cache
- kernel:sched: LOAD_FREQ (4*HZ+61) avoids loadavg
- readahead edits
- ext4 edits for smoother rom
- init.d Supported
- OC now Stable up to 1.44GHz
- Live OC is limited to 115%
novic_dev: http://forum.xda-developers.com/showthread.php?t=1525257
- I/O Schedulers: cfq, bfq, deadline, sio, noop, V(R) (Default is V(R))
- CPU governors: minmax, lazy, lulzactive, lagfree, interactive, interactiveX, conservative, ondemand, userspace, powersave, smartass, smartassv2, scary, performance, brazilianwax, intellidemand, lionheart, ondemandx, savagedzen, Wheatley (Default is ondemand)
Linux Kernel version 3.1.10
GRSecurity version 2.2.2 with different security features enabled by default (see GRSecurity Working Feature List for details)
OC up to 2.0 GHz (still need to be tested!!!)
Advanced LiveOC!!! (ALOC) support with Selective LiveOC MOD (thanks to byeonggonlee) - use this feature with caution now due to LiveOC unlimited up to 150% for testing purposes!!!
EXT4, NTFS, CIFS support
init.d support
Compatible with any OTA-based and AOSP ICS Roms
Can be managed with NSTOOLS
Provides Galaxy Nexus boot animation
Voodoo Sound and Vodoo Colors
BIGMEM for more available memory
Backlight Notification(BLN, enabled by default with blinking)
Backlight Dimmer(BLD, disabled by default)
Battery Life Extender (BLX, not active for now, need testing)
TouchWake (TW, enabled by default)
Custom Voltage Interface
Deep Idle version 2 (thanks to Ezekeel, disabled for now due to stability issues!)
CPUIdle from Linux 3.2.x backport
SLQB Memory Allocator with GRSecurity integration (using SLUB for now due to stability issues!)
USB OTG (Build 5) support (use it with this app)
Blue Colors support for Screen Color Scheme
50 FPS and 65 FPS Screen Refresh Rate support
Franciscofranco: http://forum.xda-developers.com/showthread.php?t=1357333
- I/O Schedulers: deadline, noop (Default is deadline)
- CPU governors: interactive, conservative, ondemand, performance (Default is ondemand)
- 3.0.8 kernel version
- Full ramdisk with the proper mounting flags for extreme speed
- tweaked Conservative governor
- SLQB memory allocator
- Kernel stripped of debug options +others like cgroups, autogroup, swap, etc.
- TinyRCU as being the obvious choice
- CRC32 lib updated
- I/O-less dirty throttling (backport: kernel 3.2: http://kernelnewbies.org/Linux_3.2)
- Ext4 with 3.2 patches (see link above)
- Proportional Rate Reduction for TCP (see link above)
- Fsync() to oblivion just like I had in the GB kernels
- Init.d support in the ramdisk
- BLN
- Voodoo Sound
Peter Alfonso: http://forum.xda-developers.com/showpost.php?p=21363787&postcount=265
- I/O Schedulers: cfq, deadline, noop (Default is cfq)
- CPU governors: conservative, ondemand, userspace, powersave, interactive, performance (Default is ondemand)
- https://github.com/peteralfonso
- Optimised Stock Kernel
Thalamus (Part of Oxygen): http://goo.gl/o9bxK
- I/O Schedulers: deadline, noop (Default is deadline)
- CPU governors: ondemand, performance (Default is ondemand)
- Optimised Stock Kernel
Link to stock ics kernel: http://forum.xda-developers.com/showpost.php?p=21439719&postcount=120
This is nice. Also, it could be a good idea for roms. The only thing is to redo the benchmarks every once in a while, as the developers keep updating. Great work!
Transmitted from somewhere in space... from my Nexus S... and Tapatalk.
Steve Garen's kernel seems to have the best benchmark results. Anyways thanks for this I was curious about this but too lazy to do it myself.
Sent from my Nexus S using Tapatalk
So it's a test for the performance of kernels against Slim ICS ROM.
Many thanks to krarvind for Slim ICS, the host ROM.
Testimonials
Kernel developer
mathkid95 said:
Good job with this! It can be very helpful for ppl wanting to see how the kernels stack up...
Click to expand...
Click to collapse
Kernel developer
eugene373 said:
It's nice to see somebody put raw facts behind something & not just bloated scores from Quadrant..
Click to expand...
Click to collapse
XDA user
mobile_pc said:
Your great research inspires the kernel devs to further efforts
Thanks
Click to expand...
Click to collapse
Rom developer
Cybergr said:
Things are getting more clearer now..thanks for taking your time testing and posting such a useful comparison of different ICS kernels...Helps everyone who is really intrested and sometimes confused...devs or not...but specially the lazy one! Very cool and interesting idea to post a Kernel comparison in here bedalus..!!
Click to expand...
Click to collapse
XDA veteran and kernel crackflasher
simms22 said:
... when things differ and deviate from what you expect that draws attention. thats why benchmarks can be very useful to developers. they will know right away if they messed something up or did something very right... impressed with your effort. thank you
Click to expand...
Click to collapse
Kernel developer
steve.garon said:
Cool comparison. Tells me that I can improve 2D and 3D score somehow...
My crackflashing/crackbuilding habbits are surfacing again.
Click to expand...
Click to collapse
suksit said:
So it's a test for the performance of kernels against Slim ICS ROM.
Click to expand...
Click to collapse
I'm not certain, but I believe the performance of the kernels may be compared directly so long as the testing circumstances remain absolutely equivalent.
Very cool to see.. Nice work bedalus
Thank you!
bedalus said:
Oh no, what have I let myself in for!
Click to expand...
Click to collapse
You're in for a lot of work, but I don't think it's necessary to update after each of the kernels in updated, just on a periodical base (of course, you have a life and that comes first)
Cool comparison. Tells me that I can improve 2D and 3D score somehow...
Be careful when you read IO scores. There are some tweaks to improve IO that give better benchmark result but poor actual performance while loading app. Benchmark don't tell the whole story.
bedalus said:
I'm not certain, but I believe the performance of the kernels may be compared directly so long as the testing circumstances remain absolutely equivalent.
Click to expand...
Click to collapse
I just think there may be some more variables to control e.g. SG kernel runs at 1.1GHz by default
steve.garon said:
Cool comparison. Tells me that I can improve 2D and 3D score somehow...
Be careful when you read IO scores. There are some tweaks to improve IO that give better benchmark result but poor actual performance while loading app. Benchmark don't tell the whole story.
Click to expand...
Click to collapse
Thanks, I'll add that to the information in the notes. Cheers Steve!
Great charts and scores, thank you
Cool work u have put in buddy, keep it up.....!!! Cheers....!!!!
Great work thank you so much for this Benchmark. Adding battery life comparison will make it absolutely perfect but I think it will take a lot of time to be significant (minimum 2 hours for each Kernel at the same conditions I think).
Thanks again for this good work man
tchaari said:
Great work thank you so much for this Benchmark. Adding battery life comparison will make it absolutely perfect but I think it will take a lot of time to be significant (minimum 2 hours for each Kernel at the same conditions I think).
Thanks again for this good work man
Click to expand...
Click to collapse
Yeah... Adding battery life would just take forever and there is too many variables.
steve.garon said:
Yeah... Adding battery life would just take forever and there is too many variables.
Click to expand...
Click to collapse
Yes but it is not that difficult to do. I am thinking about a full battery charge, flash Kernel and let it idle for 2 or 3 hours. Than just note the remaining battery charge.
Full recharge again and the next Kernel... Maybe just doing it on 3 kernels takes a complete day
Do any of these benchmark apps actually support ICS?