Database Users

[app idea] ROM benchmarking app

Hello guys,
I would like to ask if there is an app that would benchmark the ROM performance as opposed to the device performance as in the most of bencmarking apps available (SkTools, SPB Benchmark...). These apps can be used to benchmark ROMs too - but they show more about actual drivers performance than real life experience.
I did a search on the Internet but found nothing.
Is there anyone able to code such an app for the xda-devs community?​
I am looking for program that would measure following:
- Boot up speed
- Start menu open speed
- Speed of opening settings menu from the Start menu
- Speed of opening programs menu from the Start menu
- Speed of opening the Windows folder from the Explorer
- Speed of opening a standard built in app (for example emulating green phone button press to open the dialer)
- Speed of changing screen orientation form portrait to landscape and back
- Speed of wake up from stand-by
- catch-all master statistic (derived from all other stats) to make it easier to rank the results for various ROMs/Devices
Any other statistics you may consider important when it comes to the speed of your shiny new cooked ROM. Please advise on what can also measure actual ROM performance.
The app would ideally run these tests several times (10x) and then log the average values into a results file.
I think it would help the ROM cookers to troubleshoot their creations and new users to choose the right ROM for them. In short this new application would test the GUI reactions - no benchmark app currently available does it. It is an important characteristics though, because that is what users come in contact with most often.
Thank you for your contributions!​Thanks to MaRaHoX for his advice

May we will take a look ...

CharlyV said:
May we will take a look ...
Click to expand...
Click to collapse
Thanks a lot. I think GUI performance benchmarks are useful even in device to device comparison.

(Rare post by me but...)
Perhaps a single benchmark score would be good to determine the relative difference between the ROM being tested and a controlled test....ie WM6 Vanilla??
so a ROM which scores out in total maybe 3 times better (overall) than the control ROM would score like this:
"XDA-Bench score: 3.00" .............. something like?
the best overall performing ROM would have the highest overall score relative to the control test (vanilla)............im not sure im explaining my concept very well?!?!
ofcourse all the test results could be viewed in detailed view in the same sort of format as the xls spreadsheet??
This way a ROM can undergo a test with a custom setup on anybodys device and get a score to really see how it fairs against other ROMs? I personally see this as a similar sort of thing as PCMark or 3DMark??
EDIT::: just thought it would be cool to see how an app installed affects the XDA-Bench score?? Ive noticed certain apps tend to slow down the device whilst other dont?!

mrvanx said:
(Rare post by me but...)
Perhaps a single benchmark score would be good to determine the relative difference between the ROM being tested and a controlled test....ie WM6 Vanilla??
so a ROM which scores out in total maybe 3 times better (overall) than the control ROM would score like this:
"XDA-Bench score: 3.00" .............. something like?
the best overall performing ROM would have the highest overall score relative to the control test (vanilla)............im not sure im explaining my concept very well?!?!
ofcourse all the test results could be viewed in detailed view in the same sort of format as the xls spreadsheet??
This way a ROM can undergo a test with a custom setup on anybodys device and get a score to really see how it fairs against other ROMs? I personally see this as a similar sort of thing as PCMark or 3DMark??
EDIT::: just thought it would be cool to see how an app installed affects the XDA-Bench score?? Ive noticed certain apps tend to slow down the device whilst other dont?!
Click to expand...
Click to collapse
Thanks for your ideas - Available benchmarks have something like synthetic master score (check my benchmark thread - link is in my sig). Slowdown measuring is also interesting idea. Hopefully somebody will make this xda-bench a reality

Testing methodics
Hi KarhU,
I was thinking about it and one more thing about it - it is also really important to define starting point for this ROM - or better is to say "comparable state"... f.e. - to switch off all today items etc. I understand that for user is better to know how the ROM is powerful in real state... maybe to consider this note as a basic for discussion about it (if it is not already mentioned there or somewhere else )
Glad to help little bit with this important and purposeful work!
M.

I think the official HTC rom would be the best to use as a control rom for comparisons. I would run this app every time I install a new piece of software, at the moment my orbit is running a bit slow, I've just overdone it a bit at the moment, so it would be great to see exactly how an installation affects my system. It would be useful to developers and cookers alike. I hope the idea really takes off.

This would be a great app if developed! I agree with a those above that the official HTC rom would be the best one to use as a control rom for comparisons.

MaRaHoX said:
Hi KarhU,
I was thinking about it and one more thing about it - it is also really important to define starting point for this ROM - or better is to say "comparable state"... f.e. - to switch off all today items etc. I understand that for user is better to know how the ROM is powerful in real state... maybe to consider this note as a basic for discussion about it (if it is not already mentioned there or somewhere else )
Glad to help little bit with this important and purposeful work!
M.
Click to expand...
Click to collapse
Interesting idea - currently my tests were done WYSIWYG method - I wanted to compare the ROMs with their basic settings to see actual performance in more "real life" conditions. You are right that if we want to compare the base there should be comparable testing starting point.

I suggest to add the following test for phone devices:
- time lag between 'caller heard first ring' and 'the device actually rings'

ahlok_hk said:
I suggest to add the following test for phone devices:
- time lag between 'caller heard first ring' and 'the device actually rings'
Click to expand...
Click to collapse
This is a good one and very often people are interested in this statistics, but it is hard to measure as you need two phones.

::bump that!::

Bump! Time to resurrect the thread

i like this idea

Linpack scores for every custom ROM

Do you run a non-sprint ROM?
Help us out by booting your phone without SetCPU enabled and post your linpack score* on the wiki:
http://forum.xda-developers.com/wiki/index.php?title=HTC_Supersonic/ROMs
Make sure your version matches the version posted before posting your score.
On behalf of the community, thanks!
* Linpack scores are not the be-all and end-all of the value of a ROM. Results may vary from device to device. If you experience an erection lasting more than four hours, consult your physician immediately.

Why not post Quadrant scores too?

by all means, add a column for quadrant.
I just used linpack because i wanted to keep it simple. But if you want to add quadrant scores (or any other benchmark) go ahead

I like this idea Quadrant column added

Added scores for Fresh - using HTC's current kernel. Don't kernels effect a roms performance; could skew each person's results, right?

The problem with this set up si that the scores very to much from device to device and that depending on how the phone is set up etc or how the test is run determines the outcome.
We saw the same issue when neif was doing his rom comparison. some where taking averages some where taking best some where using everything stock some werent even with the test instruction unless one person does all the tests exactly the same for each rom on the same phone with the same set up .... it makes the numbers very useless in my opinion. Thats not even accounting for diffrent rom versions kernels etc.
So i would use this with a big grain of salt as there is basically no standard used in obtaining it if it is done as a community and YMMV.

omegasun18 said:
The problem with this set up si that the scores very to much from device to device and that depending on how the phone is set up etc or how the test is run determines the outcome.
We saw the same issue when neif was doing his rom comparison. some where taking averages some where taking best some where using everything stock some werent even with the test instruction unless one person does all the tests exactly the same for each rom on the same phone with the same set up .... it makes the numbers very useless in my opinion. Thats not even accounting for diffrent rom versions kernels etc.
So i would use this with a big grain of salt as there is basically no standard used in obtaining it if it is done as a community and YMMV.
Click to expand...
Click to collapse
+2 THIS for the sake of good science!!!!
Who knows how many variables you are not taking into account!
If you really want to do this right, you need to ensure that the ROM is the SINGLE VARIABLE for the tests.
Now who wants to take the time to do that???

I have never even remotely tried to update a wiki page, but if someone wants to post my score, running Fresh ROM 3.3.0.1 with the Netarchy 4.1.9.1 BFS kernel I got 35.523 Linpack.

[REQUEST] Antutu Benchmarking from Locked/Stock/Stock or variations

Hi. Title says it all.
If anyone has a locked bootloader, stock ROM, stock launcher, (Basically new out the box), I'd love to see your AnTuTu benchmarks.
Also, I'd be interested in any other combination of unlocked/locked bootloader, STOCK ROM, and stock or 3rd party launcher. (If not stock rom, please post your BMs and details in the threads for your ROM )
THANKS! In advance (and when you post your screen). :good:
---
https://play.google.com/store/apps/details?id=com.antutu.ABenchMark
If you're new and want to help out, (because let's face it, who else has locked/stock/stock?) what you'll do is click that link above, choose to install, send to your Nexus 7. After a couple minutes or when your N7 notifies you that AnTuTu benchmark is installed, open the device, click the 2nd tab at top called "TEST" and then hit "Start Test".
This will take about 5 minutes. You'll get a benchmark screen but go ahead and give it a few more minutes, in fact on the first benchmark screen you see, (with a score at the top) if you tap the first divider labelled "Detailed scores", you'll see that more parts of your system are still being benchmarked. Make sure to pull down this DETAILED SCORES before taking the screen screen!
After that, and all benchmarking is done, hold power+vol down to take a screen shot. You can then post from your N7 to here or send it to yourself/etc to post the screenshot here.
Attached is an example (from my N7, which is unlocked/SGT7 ROM/modified stock launcher) of what it should look like.

Locked or unlocked has zero effect on performance/becnhmarks/battery etc. Effects nothing at all.
Sent from my Nexus 7 using Tapatalk 2

Thanks for your valuable input. albundy.

Thanks for the sarcastic reply after making a thread with such a silly idea to bench with a locked and unlocked bootloader.
It makes no difference. I am curious to how you can even ponder the idea that it could to be honest.
If you where so curious about it do it yourself. Waste your own time instead of trying get a noob to waste theirs. Anybody that isn't a complete newbie knows already it makes no difference and most would even know why.

albundy2010 said:
Thanks for the sarcastic reply after making a thread with such a silly idea to bench with a locked and unlocked bootloader.
It makes no difference. I am curious to how you can even ponder the idea that it could to be honest.
If you where so curious about it do it yourself. Waste your own time instead of trying get a noob to waste theirs. Anybody that isn't a complete newbie knows already it makes no difference and most would even know why.
Click to expand...
Click to collapse
I'm looking to gather benchmarks from a variety of configurations with stock ROM - that's the goal. The bootloader is quite obviously the least important part of this. I included it because there are quite a few people who may have unlocked and rooted and don't realize that it doesn't matter and to avoid filling the thread with meaningless posts that don't at all provide to the conversation like "Does it matter if I have unlocked my bootloader?". Basically, I was providing the detail that I would love to see benchmarking off *all* configurations with stock ROM.

antutu is mostly cpu speed dependant, with some infuence on 2d/3d scores if or if not you have force gpu acceleration enabled in developer options(device main settings). mostly, scores with similar cpu speeds will score the same(root or stock) if you bench with that cpu speed locked(for example 1200mhz high/1200mhz low). if you let the device scale(set cpu speed to the highest high and lowest low) then you wont know what cpu speed its actually benchmarking because the device scales and decides what cpu speed it wants. allowing it to scale can produce a huge variation in scores and is inaccurate in providing the true scores for a certain cpu speed, stock or root.

[Q] Do custom roms respect Google's thermal throttling?

Hi guys, Getting delivery of my nice new N4 tomorrow and already looked out the custom roms I fancy trying but I saw some videos with people describing lagging in some games, which is most likely explained by this video (which the forum won't let me post) ...
Youtube url = abf7nPiUUE8
In it, the presenter explains what thermal throttling Google has applied to stop the phone bursting into flames and killing everyone.
My question is, as the title says, do all the custom roms respect these thermal throttling values or will I find custom rom makers changing these to dangerous values just so their rom has the illusion of much more speed and performance of other roms even though this would come at the cost of possible bricking/destroying your phone and possibly injuring you or putting your life in danger.
Sorry if I made it sound over-dramatic.

djsubterrain said:
Hi guys, Getting delivery of my nice new N4 tomorrow and already looked out the custom roms I fancy trying but I saw some videos with people describing lagging in some games, which is most likely explained by this video (which the forum won't let me post) ...
Youtube url = abf7nPiUUE8
In it, the presenter explains what thermal throttling Google has applied to stop the phone bursting into flames and killing everyone.
My question is, as the title says, do all the custom roms respect these thermal throttling values or will I find custom rom makers changing these to dangerous values just so their rom has the illusion of much more speed and performance of other roms even though this would come at the cost of possible bricking/destroying your phone and possibly injuring you or putting your life in danger.
Sorry if I made it sound over-dramatic.
Click to expand...
Click to collapse
Yes, you made it very dramatic. No dev will intentionaly build a rom or kernel to fry your cpu just to get higher clock speeds or benchmark results. Undervolting with a custom kernel will save you from some heat issues also.

gee2012 said:
Yes, you made it very dramtic. No dev will intentionaly build a rom or kernel to fry your cpu just to get higher clock speeds or benchmark results. Undervolting with a custom kernel will save you from a some heat issues also.
Click to expand...
Click to collapse
Thanks for the answer although you didn't actually answer my query, I was wondering if someone could check their custom rom and tell me if the same file is there, with the same values as Google had set, just so I can confirm if custom rom designers use the same values.

djsubterrain said:
Thanks for the answer although you didn't actually answer my query, I was wondering if someone could check their custom rom and tell me if the same file is there, with the same values as Google had set, just so I can confirm if custom rom designers use the same values.
Click to expand...
Click to collapse
Why not just look yourself? It is not that hard to DL a rom and have a look.

none of the kernels(yes, kernels not roms. roms have nothing tio do with it) disable thermal throttle, many raise the limit slightly though. but, there are many of us, like me, that chose to disable thermal throttle manually. and no, your phone wont burst into flames and explode. there still exists a safety, when you hit this safety temp of 100C, the phone will automatically turn off. yes, its safe to get that hot. and you will never ever reach that temp if you arent just trying to get your phone hot.

djsubterrain said:
Thanks for the answer although you didn't actually answer my query, I was wondering if someone could check their custom rom and tell me if the same file is there, with the same values as Google had set, just so I can confirm if custom rom designers use the same values.
Click to expand...
Click to collapse
Most custom kernels are modfied stock kernels, better check the kernel threads but i guess they follow the stock kernel directory`s. As said undervolting decreases heat considerably especialy when gaming and performing intensive tasks .

Benchmark Result of Custom ROMs (With Modifications) [not yet updated]

Xiaomi Redmi Note 4x Customised Benchmarks​"Thrive to strive the best performance possible"​This thread focuses on benchmarks and device performance which is achieved by tuning and modding the deivce.
In the two sections, the first section comprises of benchmarks which i have performed on my device, (I have the indian variant, with 32gb of storage and 3gb of ram) and the second comprises of user submitted results.
Read post 2 for details regarding submission of benchmarks.
:SECTION 1:​Benchmarks performed by me
This section will be updated soon.
:SECTION 2:​User Submitted Results
This section will be updated soon.

Rules For Submission of Benchmark
Post 2
Please make sure that
To submit a benchmark please reply to the third post saying "I want to submit a benchmark" instead of quoting this whole op.
When submitting a result, please note and report these things-
1. The Device variant which you have. (2gb, 3gb or 4gb)
2. The version of android you are using.
3. The full ROM Name and Version. (if possible then provide the download link)
4. Detailed explanation of the tweaks you applied, and their version you used, for example the name of custom kernel used, its version, what addons you are having, what extra packages like gapps or magisk you have installed etc.
5. The benchmark should be performed using both Antutu Benchmark and Geekbench 4. (if possible provide the app version)
6. When submitting either you can attach a screenshot, or write the result, but please note that to avoid any fake submissions, the screenshots are much preferred.
These results must be present -
For Geekbench 4 - Single Core score, Multi Core Score.
For Antutu - Overall score, 3D Score, UX Score, CPU Score, Ram Score.
7. Proper credits will be given to you, so don't worry

-Post 3-
i want to submit a benchmark