Okie dokie, simple plan to directly compare clean ROMs from the cooks without opinions or bias.
I believe the general idea was to either add a new table to the Wiki or modify the existing tables in the ROMs section, to display standardised benchmark result, ie all from the same version following a set of instructions
those might be, flash ROM, hard reset, install bench mark programs, softrest, softrest run benches with radio on/off
Could be expanded to include basic battery life i dont think having call time / sms count would be helpful as its too unpredictable. but perhaps standby time and or wifi/bluetooth turned on but not connected.
I dont think its about real world tests since we cant have opinions, its a simple OS comparison.
course, for all i know there will be almost nothing between any of them making it redundant so this is why im making this thread, to talk it out and see if its worth while.
and you are starting this "new" thread as a result of my ideea and proposal
wouldn't have been nice to mention?
Benchmarking will give you a very incomplete picture of how a rom is actually going to perform, and therefore will be a virtually worthless use of your time. The real performance of a rom is gonna very significantly based on what software you use, which versions of the software you use, and sometimes even the order in which you install it. There is no consistent reliable benchmarking tool that i have seen as any useful indicator of actual speed of a rom.
The best way to find out about what rom will suit your needs is to read the frist couple posts, then kinda browse through the thread to see which bugs people are posting most regularly. When i was testing the WM6.5 roms, I had no issues with the fact that some people had exchange issues, because I was not having them, but eventually the fact that the notifications weren't working correctly caused me to change to something else. I tried another rom with some TP2 features, and generally liked it, but MyPhone didn't work correctly on that roms for whatever reason.
The point I'm trying to make is that there is no such thing as an unbiased benchmarking, because if we only test them all with no software installed, running clean then you're gonna find that almost all roms score similarly. Even more to the point, I simply have never seen a benchmarking program which had results which were indicative of any real world performance, and as such, I have disregarded the use of them entirely.
If you're happy with the rom you're on, then keep it. If not then read a few threads, see what is out there and then based on a little bit of research try one. If you don't like it, then try another one or even go back to the old one.
noris08 said:
and you are starting this "new" thread as a result of my ideea and proposal
wouldn't have been nice to mention?
Click to expand...
Click to collapse
Sorry, looking after my 3 year old today, pulling my hair out, damn cheeky monkey!but yes totally based on your idea, just wanted to make the link in the old post before it got the chop ill edit the original post in a min
scotchua said:
Benchmarking will give you a very incomplete picture of how a rom is actually going to perform, and therefore will be a virtually worthless use of your time. The real performance of a rom is gonna very significantly based on what software you use, which versions of the software you use, and sometimes even the order in which you install it. There is no consistent reliable benchmarking tool that i have seen as any useful indicator of actual speed of a rom.
Click to expand...
Click to collapse
amen to that!
just making a little bit of reading and got scared by the possibility of threads about "witch ROM is better" beeing aloud
this is going to end badly as i already know that from the trinity forum and from the diamond forum. the forum is going to be cluttered only because a few lazy sobs are not ready to spend a few moments reading the ROM's threads and drawing their own conclusions. it is enough to read the first page, one of every 3 pages and the last one in a ROM thread to understand what is all about.
not only that but when the thread is going to be too long some smart ass is going to open another (he has no time to read a l l those pages)...and than another. please, stop the madness
scotchua said:
The real performance of a rom is gonna very significantly based on what software you use, which versions of the software you use, and sometimes even the order in which you install it. There is no consistent reliable benchmarking tool that i have seen as any useful indicator of actual speed of a rom.
Click to expand...
Click to collapse
I agree, to compare ROMs completely requires opinions and subjectional review but since ive not tried this idea on different ROMs i couldnt say for sure what the results would hold.
Once upon a time, before the internet age, there were "Informatics" and "personal computers"
We had a problem: every shop sell HIS PC, assembling HW and assessing "My PC is the faster one". Also many TWEAKS were proposed for DOS and applications (This was before Egyptians I believe...)
The solution?
We built a SUITE, using a scripting tecnique, EMULATING REAL LIFE USER and measuring time, memory usage, battery load, CPU load...
Instead of using commercial benchmarks, because we don't care really about the file system speed, scrolling a large DIR matters, or rotating the screen speed.
We emulated a PRO user and a FUN user, opening a standard office file, starting and closing TT7 or video player speed with standard app (es. TCPMP) , opening an IE page, internet speed, scrolling a LOOOONG contact list, anything you consider as normal life usage.
A script could be used to start every single task and measuring the time.
AND IMHO THIS will be the killer application, comparing different ROM, devices, Diamond is faster than HD? and Kaiser? WVGA how slower is in REAL LIFE?
i belive you miss the point
this is supposed to be a tool helping people with less time or experience to choose the rom that is more appropiate to them
and maybe even help the developers to improve their work
as in the original thread is mentioned it isn't ment to say which rom "rules"
but, as i already said, if people will find this useless, or discussion will degenerate, a mod can close the thread anytime
noris08 said:
i belive you miss the point
this is supposed to be a tool helping people with less time or experience to choose the rom that is more appropiate to them
and maybe even help the developers to improve their work
as in the original thread is mentioned it isn't ment to say which rom "rules"
but, as i already said, if people will find this useless, or discussion will degenerate, a mod can close the thread anytime
Click to expand...
Click to collapse
I don't think we missed the point, just trying to share from experience why benchmarking isn't a particularly useful gauge to users trying to pick a rom. If you don't have the time to look around then i'd actually suggest just picking a rom based on what you read in the first couple posts describing it. Also that type of users should just pick a rom that is fuller with more programs installed by default.
Great idea, but will it ever be real?
While it's possible to devise a set of benchmarks and some kind of subjective scoring algorithm, the measurement accuracy will depend on too many factors and the result is likely to be unreliable and inconsistent. Seeing how many various ROMs based on the same OS builds are getting very different ratings in ROM benchmarking threads, real life performance seems to depend on every modification implemented in a ROM. E.g. even if the difference between ROMs is merely in a few files or a few registry keys, you'll probably end up comparing apples to oranges so this won't be very useful.
stepw said:
Great idea, but will it ever be real?
While it's possible to devise a set of benchmarks and some kind of subjective scoring algorithm, the measurement accuracy will depend on too many factors and the result is likely to be unreliable and inconsistent. Seeing how many various ROMs based on the same OS builds are getting very different ratings in ROM benchmarking threads, real life performance seems to depend on every modification implemented in a ROM. E.g. even if the difference between ROMs is merely in a few files or a few registry keys, you'll probably end up comparing apples to oranges so this won't be very useful.
Click to expand...
Click to collapse
I must agree to that! At the end of the day, it all comes down to what started this thread: the personal feeling a user gets when using a ROM in his own particular way of using it, which is totally different and uncomparable with other's.
it seems that i can not make you see my point
i will say it for the last time
my original ideea was NOT to compare one rom to another rom.
that will only lead us to square one - which rom is best
AND THIS IS NOT THE POINT!
whatever!
i'll give up
cheers!
tnyynt said:
I must agree to that! At the end of the day, it all comes down to what started this thread: the personal feeling a user gets when using a ROM in his own particular way of using it, which is totally different and uncomparable with other's.
Click to expand...
Click to collapse
Benchmarking suite....
1) Free memory, storage mem at first start
2) Boot up time (measured with a simple program started at the end of the boot)
3) Search a string (time)
4) Open a complex word document (time)
5) open a complex excel document (time)
5) Active sync connected, open a simple web page, time, open a a Complex web page (success, time) (IE)
6) Open a LOCAL complex web page, scrolling, (time)
7) Restoring 4000 contacts (pimbackup, success, time)
8) Restoring 2000 SMS (pimbackup, success, time)
9) TT7 startup and close (or other BIG software, time)
10) TCPMP standard video player performance (direct draw, accelerated)
3 times each test, % of battery resulting startin with a full charge (these are only examples, just to explain better the point)
noris08 said:
it seems that i can not make you see my point
i will say it for the last time
my original ideea was NOT to compare one rom to another rom.
that will only lead us to square one - which rom is best
AND THIS IS NOT THE POINT!
whatever!
i'll give up
cheers!
Click to expand...
Click to collapse
i agree to noris08!
this thread is not what was the original intention
i doen't want to know if some rom is about 10ms faster than other rom as i have readed in other threads where one published his benchmark score 2635. and a user posted the question if his device fails because he comes "only" to score 2450 e.g.
no, for example i testet 5 ROMs from 4 chefs.
i would be able to tell the reason for choosing the actual ROM on my HD. There where many facts hardly subjective opinions for choosen that! I think if a user would describe his decision respectfully to the hard work of the cockers this will help all of us.
if the guy is wrong other users can reply with their opposite experiences he knows it is worth to look for the reason on his device(configuration).
We all know speed, good feeling depends on many factors. so only a respectful discussion will help users and chefs.
even when user are telling a specific ROM is slower than other specific rom, other users can agree or disagree. if one is telling it is slow and many others reply the opposite this is a useful information for the user and the chef. chef get the response his rom seems to work fine. otherwise the chief get the information his rom leaks in speed and there are further tuning possibilities because other rom look to be faster.
I would find a thread e.g.
"Your experences with different ROMS" or
"what rom do you use and which property or feature do you like extra"
maybe useful.
If someone is posting: "ROM X ist the best" useres are old enough to know that this is not a meaningful post.
I would like to read postings like:
I am using ROM XY - has great looking german keyboard with äöüÄÖÜß - very stable (softreset max. 1 in 2 Weeks ) Battery: without Backligth and no running program and no dataconnection only 60mA i experienced a "normal" value, in suspendmode over night max. 1% Power loss although G-Alarm and phone active ) overall good responses.
are you guys so scared of whos rom will be the fastest??
end of the line the fastest rom will be the one better built. (the one more tweaked)
its like talking to ppl :
"wow its stupid comparing a dell pc to a alienware pc with the same hardware"
why??
its like if someone is a fanboy of ati and never admits a nvidia car will perform better & vice-versa
and of course as like the nvidia & ati cards, not only hardware will make a difference.
do you want to know how it really should look a ROM review?! did you ever imagine how complex a ROM is to build a ROM? do you know what is the difference between a good and a bad ROM? here u have an example from one expert i'll trust with an opinion!
the-equinoxe said:
Cooking a ROM isn't just trowing a few packages, and some registry fixes in a kitchen and press build..
It's knowing why a ROM behaves in a certain way, and fixing unwanted behaviour.
It's knowing what happens when you cook a ROM, what the scripts in your kitchen are supposed to do, and what the are actually doing(!!).
It's knowing this and so much more..
I have seen ROMs released lately where the XIP wasn't rebased, where the rgu's were contradicting the hv files and worse: where RGU from package a was contradicting the RGU from package B. (and even worse: A contradicting B but both contradicting HV..).
Why not simply import the RGU's into the HV files first? and check the rgu files, or for fraks sake, merge them!
Some had added certificates, but the cook didn't knew that the base he was using was already patched to ignore all certificates.
Why on earth clutter the ROM with unnecessary certificates? Really WHY?
I have seen cooks adding XIP of a higher build but using the OS of a lower build, just to get a high build number. (some just plain hexedit the build number :s )
This simply makes me puke, why on earth would you add an UNMATCHING XIP in another OS? You would think that that unmatched part was the cause of some unexpected bugs, wouldn't you..
Or massive amounts of files that are moved from root (=\\windows) to some kind of subdirectory, it seems that the chefs who are doing this are unaware that they aren't MOVING but COPYING the files (jups, that was bad design from microsoft)
Why concentrate on the build number?
A higher build number doesn't make a better ROM.
Focus on making a ROM BETTER, fix those contradicting registry entries, there are plenty of tools out there to ease your work nowadays, it can be done in hours instead of weeks.
Rebase those files that are supposed to be XIP, don't leave it unaltered and most certainly don't make it a PE-file (like dll or EXE), you will have unpredictable occurrences of drivers unloading from memory when doing this, and other crazy bugs that are hard to pinpoint.
And if another chef removes those dsm files and replaces all RGU's with one (or just simply stay with the HV) ,it's mostly not to piss off other chefs so it can't be shared, but to make the ROM better and faster.
Etc etc etc etc.. (I could go on and on.. really ! The curse of the kitchens I called it)
This is not a Flame to a certain Chef, or even directed at this particular Forum, I have many devices, and I have flashed them a lot, sometimes I take the time to analyze a ROM, and I have been amazed what junk has been produced by some..
The main idea of a cooked ROM is to have a better device with fewer bugs, not a fancy picture, nor to have the highest number..
So in some cases the cure is worse than the problem..
I am not going to single out a bad cook, nor a good cook, there are plenty of both..
Some will see this as a personal attack, personally: I don't care, if you claim to be a Top-chef, but don't know the basics of cooking, call yourself what you want, but don't expect me (or others) to be fooled.
I am just saying: instead of focusing on a high build number, or a (bug ridden) beta, focus on what you are actually doing.
Take a stable ROM, and make that one better, most AKUs are intended to support newer devices, and newer hardware, a higher AKU doesn't mean the ROM is better (it could have new bugs to deal with).
Click to expand...
Click to collapse
the whole post is here:
http://forum.xda-developers.com/showpost.php?p=3387292&postcount=76
KukurikU said:
did you ever imagine how complex a ROM is to build a ROM?
Click to expand...
Click to collapse
sorry, but this does not subject this thread and nobody doubts about ROM cooking is not a simple job! thanks to all chefs!!!!
KukurikU said:
do you know what is the difference between a good and a bad ROM?
Click to expand...
Click to collapse
from the user's viewpoint i think its clear and software is made for users. user do not mandatory want to know whats behind the scenery in deepth. user want a device with software with basic properties and naturally many cool features - lets say:
a stable ROM with all features he needs for daily use, cool features for fun and suitable speed. if the userinterface have a appealing design also it's nearly perfect!
and here we are back again:
"what features or properties do you like most"
would help users to find out, what is the best for him!
this is naturaly only my opinion
P.S: Sorry for my suggestion to open a thread, i just found a thread with such a Subject!!!!!
autdev said:
sorry, but this does not subject this thread and nobody doubts about ROM cooking is not a simple job! thanks to all chefs!!!!
from the user's viewpoint i think its clear and software is made for users. user do not mandatory want to know whats behind the scenery in deepth. user want a device with software with basic properties and naturally many cool features - lets say:
a stable ROM with all features he needs for daily use, cool features for fun and suitable speed. if the userinterface have a appealing design also it's nearly perfect!
and here we are back again:
"what features or properties do you like most"
would help users to find out, what is the best for him!
this is naturaly only my opinion
P.S: Sorry for my suggestion to open a thread, i just found a thread with such a Subject!!!!!
Click to expand...
Click to collapse
cool features and appealing design in a rotten ROM means high battey consumption, frequent freezes and soft resets and so on...
i think i wasn't clear enough in my previous post. what i meant (in a nut shell) was that because of it's complexity deciding that a ROM is good, less good or bad is a professional's job. only an expert can give you a trustworthy opinion.
the moment somebody starts a "what is the best ROM for me" everybody will jump in and push the ROM he is using. after a while the thread will be cluttered with hundreds of honest but unprofessional opinions and you'll have on one page 8-9 different opinions. does this make yr decision of choosing a ROM easier?? i don't think so! so, back to square one
in my wet dreams i see a thread filled with posts of "ROM critics" that are analyzing each new ROM. reading such reviews could be very useful indeed.
in short: if u are a chef and own a kitchen than beware of the "food" critics that are visiting yr restaurant. they can kill yr business in tomorrows front page ROM review or make you a very, very rich and famous chef
ok So to conclude,
we cant use benchmarks since the difference between them would be next to nothing
we cant judge a ROM based on its funtionality since its purely opinionated
we could potential use a script to run a series of real world tests that become useless after a few resets and number of programs installed
That pretty much knocks the idea on the head then?
dazza9075 said:
ok So to conclude,
we cant use benchmarks since the difference between them would be next to nothing
we cant judge a ROM based on its funtionality since its purely opinionated
we could potential use a script to run a series of real world tests that become useless after a few resets and number of programs installed
That pretty much knocks the idea on the head then?
Click to expand...
Click to collapse
sure looks that way.
I would all come down to every user deciding on his own. Would you believe that after trying lots of ROMs I've reverted to a certain Stock ROM because I find that it offers close to perfection for my needs? I could praise the speed and stability and I'd advise you to use it, but you 'd most probably find that it would not suit your needs.
Anyways, I've offered to support a thread that is civilized and based on factual opinions, I will keep my word in doing so, if there's be such thread.
Here's another idea for you all: a good factor to take into consideration when judging a ROM is the number of users using it. It's a good indicator of the ROM's value, since X no. of users are hangin' on to that. Why not post and maintain a simple poll with the most common ROMs (stock and cooked) and see where it gets you?
Hi all,
I am gearing up to get a Nexus 4 in a couple of weeks following its price drop, and I think I'd like to install a custom ROM (and maybe a custom kernel) as I like the idea of improved performance and having my OS specifically tailored to my phone. It seems like there is a massive number of options available to people who want to customize their Android phone, and I'd really appreciate some advice as to where I should start looking.
I have read up a bit on prominent ROM's like CM, PA, SlimBean, MIUI, etc, but user experiences tend to vary pretty widely. Basically, I am all for minimalism, stability, and trying to get my battery to last as long as possible. I'm really just looking for an elegant, intuitive ROM that is fast and efficient and doesn't need too much running on the back end to do what it needs to do. I really like PA's Halo notifications, but from what I have read the ROM itself can be a bit buggy and sluggish relative to other ROM's. Is there any way I can integrate Halo notifications into a more lightweight ROM?
Finally, it seems like running custom kernels has the potential to improve performance/battery life/etc, but usage is so user-specific that it's impossible to say which kernel is "best" for such things. If that is the case, is there an easy way of understanding their strengths and weaknesses so I can decide which one is right for me? I'm just a bit lost as of right now .
Thanks for the help!
Downloaded7 said:
Hi all,
I am gearing up to get a Nexus 4 in a couple of weeks following its price drop, and I think I'd like to install a custom ROM (and maybe a custom kernel) as I like the idea of improved performance and having my OS specifically tailored to my phone. It seems like there is a massive number of options available to people who want to customize their Android phone, and I'd really appreciate some advice as to where I should start looking.
I have read up a bit on prominent ROM's like CM, PA, SlimBean, MIUI, etc, but user experiences tend to vary pretty widely. Basically, I am all for minimalism, stability, and trying to get my battery to last as long as possible. I'm really just looking for an elegant, intuitive ROM that is fast and efficient and doesn't need too much running on the back end to do what it needs to do. I really like PA's Halo notifications, but from what I have read the ROM itself can be a bit buggy and sluggish relative to other ROM's. Is there any way I can integrate Halo notifications into a more lightweight ROM?
Finally, it seems like running custom kernels has the potential to improve performance/battery life/etc, but usage is so user-specific that it's impossible to say which kernel is "best" for such things. If that is the case, is there an easy way of understanding their strengths and weaknesses so I can decide which one is right for me? I'm just a bit lost as of right now .
Thanks for the help!
Click to expand...
Click to collapse
Personally, I've gone back to stock because of stability and battery life. Best thing is to try them. You can't really brick a Nexus
Be quick to pick one up. They're going out of stock very quickly.
Sent from my Nexus 4 using Tapatalk 4
NathanBookham said:
Be quick to pick one up. They're going out of stock very quickly.
Sent from my Nexus 4 using Tapatalk 4
Click to expand...
Click to collapse
yes they are. 8gb are gone.
Any word on supply scarcity in the UK? I am in Denmark right now and I can't order from the Play Store until I get back to England. If by the time I get back they are all gone, I might as well skip the wait and buy one for a little more money on Amazon right now.