Related
One of my coworkers has a tmobile vibrant with some lag fix according to him.. he did a quadrant benchmark right in front of me and it was showing 2500 plus everytime.. Im very curious as to what is making his phone so fast. And can it be dont to ours. Hes not running a custom rom or overclocking. Im only getting 1030 with mine clocked at 1.2ghz. Any Ideas? I couldnt get into too much details with him yesterday and I dont know whens the next time ill see him..
If you were to look at a test break down you would see generally all the scores are identical or the epic a little ahead except in the read/write area. The scores from their read/write are just inflating their overall score. It's a issue with quadrant and how it handles its overall score. Basically it just makes the system easy to abuse/cheat. So I wouldn't worry much about the difference in your score and his.
Sent from my Samsung Epic
The reason other Galaxy S phones score high in quadrant is because of the lag fix they use. The lag fix mounts a different file system on the phone with DRAMATICALLY increases read-write times. That portion of the quadrant benchmark gets inflated beyond reason. Using this game technique, Cyanogen was able to score more than 3000 on a snapdragon phone.
All of the Galaxy S phones have the same processor. Also, quadrant is a terrible benchmark. It's the most over-quoted and abused benchmark for android phones
Ahh ok.. thats good to know.. so what would be a better benchmark to use? Linpack?
jok3sta said:
Ahh ok.. thats good to know.. so what would be a better benchmark to use? Linpack?
Click to expand...
Click to collapse
Linpack is good for measuring raw CPU processing power... but only on devices running the same version of android. Phones with 2.2 will score insanely high due to the JIT compiler. For example, a snapdragon phone with Froyo can score ~40 Mflops. A snapdragon phone with eclair scores around 7 Mflops. Does Froyo make the phone run 5-6X faster? Hell no. In some cases, the difference is almost unnoticeable to the human eye.
Here is a rundown of what I believe to be the pros and cons of various benchmarks:
Linpack
Pros:
- Good for measuring CPU processing power on the same version of Android
- Great tool for measuring the performance gain from overclocking
Cons
- Scores are boosted unreasonably by Froyo's JIT compiler on snapdragon phones
Quadrant
Pros:
- Great tool for measuring the performance gain from overclocking
- Decent tool for measuring 3D graphics performance (just pay attention to FPS, not the end result)
- Decent tool for measuring 2D graphics performance (again, look at FPS)
- The paid version ("Quadrant Pro" I believe) shows which parts of the benchmark contributed to the score. Easier to spot the inflated CPU or I/O inflation
Cons:
- I/O portion isn't valued as much as others, but can boost scores beyond reason via exploits, hacks, fixes, etc.
- CPU portion is inflated on phones running 2.2. A Nexus One is not faster than any Galaxy S, Droid X, Droid 2, etc.
Neocore
Pros:
- Good tool for measuring graphics processing power
Cons:
- Graphics are not intense enough to push the power of very fast GPU's. Some phones will hit their FPS limit
- Only measures graphics processing power.
Nenamark1
Pros:
- Great tool for measuring graphics processing power
- Effects are advanced enough to show the performance of faster GPUs in relation to phones with lesser GPUs.
Cons:
- Only measures graphics processing power.
Sweet thanks for all the info man..
Agreed, this is great info thanks. I think the quadrant score is the most quoted becuase it provides a very easy to read graph built in with it for instant comparing/gratification. I guess I am gonna start going by linpack and nenamark1.
hydralisk said:
Linpack is good for measuring raw CPU processing power... but only on devices running the same version of android. Phones with 2.2 will score insanely high due to the JIT compiler. For example, a snapdragon phone with Froyo can score ~40 Mflops. A snapdragon phone with eclair scores around 7 Mflops. Does Froyo make the phone run 5-6X faster? Hell no. In some cases, the difference is almost unnoticeable to the human eye.
Click to expand...
Click to collapse
Linpack is ok for when your using same CPU comparison, different CPU's can cause issues...The reason why snapdragon gets scores of 5-6x is for some reason the snapdragon utilizes the VFP rather then using raw processing power..aka snapdragon cheats on the Linpack.
In reality our I/O scores should be a lot higher then it is as even in the Epic some of samsung's crappy file system still exists. But not as high as the lagfixed Vibrant of course.
Quadrant Pro is probably best indicator out of them all(The non-pro version is pretty much useless unless your comparing the same phone)...the con of having 2.2 show is higher is expected as it is a measure of efficiency of JIT in comparison to the current. The OS always played a role in Benchmarks so it is expected.
it can be faked by using a different partition to test on. IIRC the data partition making the speeds much faster than they should be so be careful when accepting those high scores
rjmjr69 said:
it can be faked by using a different partition to test on. IIRC the data partition making the speeds much faster than they should be so be careful when accepting those high scores
Click to expand...
Click to collapse
It is not exactly faking it..as you are increasing performance..thing is you cannot see at what it performs well at unless you see the individual scores from the Pro version....
What do you think Androidcentral was talking about when they said "Quadrant benchmark (which is about useless on a Galaxy S phone -- blame the cheaters and developers lack of fixing the I/O scoring algorithm for that one)" Are they blaming EXT2 and EXT4 lagfix or the developers of the program? Just very curious.
Sent from my SGH-T959 using XDA App
Probably both. With a loopback filesystem, Quadrant is only reading and writing to a buffer instead of the actual disk. So the numbers are insanely high compared to true R/W scores. However lagfixes like Voodoo and Z4mod are not creating loopbacks and instead are converting the actual disks to another, faster, format. So Voodoo scores are real.
FWIW, I don't really think Quadrant as a whole is useless, just the overall score. I think Quadrant Advanced is actually a very good benchmarking program.
Not useless
Just not representing real life. They are good for setting a comparison, but in real life, the phone, computer or anything else with arbitrary measurement tools does not getting the whole story.
Case in point, my computer does not have have the fastest cpu but out performs my friends with the fastest components, Why, because there is more to a computer than just a fast cpu and vdo card. Same for the phones.
The real life tests i did with a G2 and Vibrant side by side did not make any of us go WOW about either phone, but on a whole All of us who were there left with " the Vibrant performs better overall.
Anyway, they are fun to use and do generate conversation.
The benchmark. The insane score of the I/O makes the score meaningless. You can have a phone with awesome I/O and a horrible cpu/gpu and itll score higher then a much faster phone (evo,vibrant, etc).
Thanks for the answers and your opinions, guys. All very good points.
Sent from my SGH-T959 using XDA App
I just scored 2597 on a DK17 FroYo ROM (Quantum Rom v1.5 {WarpSpeed}) running at a stock 1.0 GHz. I believe this sets a record for a stock CPU on Android.
(At least it does according to smartphonebenchmarks.com) UPDATE - It's official. This is the 8th highest recorded Quadrant score and the very highest recorded stock clocked CPU score on that website.
This score was made possible by the EXT3 (EDIT - Correction, ramdisk) hack implemented in the most recent version of the ROM, released this morning.
EDIT - I'd like to note that the "lagfix" was incorporated into the ROM by the author with noobnl's assistance to prove a point; neither condone the serious use of Quadrant scores in it's current form. This hack is purely a Quadrant scores boost and does not provide real-world benefits. It is a demonstration that we should not rely upon inaccurate measurements to tell us what ROM or hardware is best.
An exploitable benchmark is no benchmark at all.
http://dl.dropbox.com/u/11825908/snap20101127_124836.png
http://dl.dropbox.com/u/11825908/snap20101127_131805.png
So just to clarify, does the ext3 hack do nothing other than improve benchmark scores? Is there any real-world improvement?
As common knowledge as the Quadrant thing is, I'm amazed people KEEP IMPLEMENTING this tweak and bragging about the results... unless there was an actual (less-drastic than Quadrant shows) improvement.
I agree that first we have to look at the big picture..while it may not help as much as the number makes it out to be, that doesn't necessary mean that it gives no benefit.
Next is this, why do people complain about Quadrant so much? you are using a FREE version..the FREE version only gives you a total number..Quadrant Pro on the other hand gives you the scores in each category..and is actually one of the best for measuring atm on android...
Electrofreak said:
I just scored 2597 on a DK17 FroYo ROM (Quantum Rom v1.5 {WarpSpeed}) running at a stock 1.0 GHz. I believe this sets a record for a stock CPU on Android.
(At least it does according to smartphonebenchmarks.com)
This score was made possible by the EXT3 hack implemented in the most recent version of the ROM, released this morning.
An exploitable benchmark is no benchmark at all.
http://dl.dropbox.com/u/11825908/snap20101127_124836.png
http://dl.dropbox.com/u/11825908/snap20101127_131805.png
Click to expand...
Click to collapse
it is true, we mounted benchmark data folder as a ramdisk/tmpfs, theres no ext3 filesystem in the rom because just for the lulz and educate the group that a lag fix won't do ****.. just to cheat a benchmark.. we were pissed at roms user thinking making rfs to ext3 will make performance better...
gTen said:
I agree that first we have to look at the big picture..while it may not help as much as the number makes it out to be, that doesn't necessary mean that it gives no benefit.
Next is this, why do people complain about Quadrant so much? you are using a FREE version..the FREE version only gives you a total number..Quadrant Pro on the other hand gives you the scores in each category..and is actually one of the best for measuring atm on android...
Click to expand...
Click to collapse
It has nothing to do with cost. It has to do with the fact that Quadrant is the most commonly used benchmark and it is deeply flawed.
Yes, the Pro version does break down the scores. Most people don't pay for Quadrant. Secondly, the I/O scores are weighted very heavily... the score nearly tripled because an I/O test completed a little quicker. There's a significant problem with that.
I've been impressed by the Epic's speed, but your scores aren't truly valid because you are "cheating"
I assume you're using the new Quantum ROM (which I also have installed) that implements a lagfix in which one moves the app data and dalvik cache away from Samsung’s embarrasingly slow internal Storage. Doing this significantly boosts I/O scores in Quadrant, thereby cheating the system. In real-life use, your system is barely performing any faster than before the exploit/lagfix
Read here for more info to see how a dev exploited this to create a 3300 Quad score (can't post links so remove the spaces):
http : // androidspin .com /2010/08/23/my-quadrant-is-bigger-than-your-quadrant/
This exploit has even been confirmed by the almighty Cyanogen himself
So is there any real benifits to this Epic 4G "lag fix"? I've seen that the "lag fix" within the other galaxy S phones actually does give real world results...
Eazail70x7 said:
So is there any real benifits to this Epic 4G "lag fix"? I've seen that the "lag fix" within the other galaxy S phones actually does give real world results...
Click to expand...
Click to collapse
Sure, it provides a speed benefit. The only thing being addressed in this thread is that, just because your quadrant scores double, it doesn't mean your device is twice as fast in actual use
Look for Dameon's post a couple of days ago in the quantum rom thread (on tapatalk..and linking sux). He talks about this exploit, but sadly (and understandably), most of the rom migrants want something to quantify expected results.. Even if bs
sent from my RAZR
Its not fair to say quadrant cheats.. rather it just may or may not be measuring performance of an operation that is ever really used on your system. In fact your system does use some fsyncs like quadrant and those will be faster like quadrant. Your system could start using fewer fsyncs though and quadrant would never know. Also your device certainly doesn't use only fsyncs..but those are very related to the lag. So basically, quadrant says the lag fix works if I'm not missing something. That's OK...but you don't need a crazy number to say that.
Bottom line... the only thing that tells how your phone performs...is how your phone performs and that's not just some little disclaimer. Hardware speed depends DRAMATICALLY on how it is used.
Sent from my SHW-M110S using XDA App
biff6789 said:
Sure, it provides a speed benefit. The only thing being addressed in this thread is that, just because your quadrant scores double, it doesn't mean your device is twice as fast in actual use
Click to expand...
Click to collapse
Actually the tweak, as noobnl states, quite specifically provides NO performance improvements to the phone. He would know because he made the hack. See the quote below.
noobnl said:
it is true, we mounted benchmark data folder as a ramdisk/tmpfs, theres no ext3 filesystem in the rom because just for the lulz and educate the group that a lag fix won't do ****.. just to cheat a benchmark.. we were pissed at roms user thinking making rfs to ext3 will make performance better...
Click to expand...
Click to collapse
Moving on...
biff6789 said:
I've been impressed by the Epic's speed, but your scores aren't truly valid because you are "cheating"
I assume you're using the new Quantum ROM (which I also have installed) that implements a lagfix in which one moves the app data and dalvik cache away from Samsung’s embarrasingly slow internal Storage. Doing this significantly boosts I/O scores in Quadrant, thereby cheating the system. In real-life use, your system is barely performing any faster than before the exploit/lagfix
Read here for more info to see how a dev exploited this to create a 3300 Quad score (can't post links so remove the spaces):
http : // androidspin .com /2010/08/23/my-quadrant-is-bigger-than-your-quadrant/
This exploit has even been confirmed by the almighty Cyanogen himself
Click to expand...
Click to collapse
This was the whole point of my post... I think you might have missed that. I would also recommend this article as well: http://briefmobile.com/cyanogen-demonstrates-quadrants-flaws
Electrofreak said:
It has nothing to do with cost. It has to do with the fact that Quadrant is the most commonly used benchmark and it is deeply flawed.
Yes, the Pro version does break down the scores. Most people don't pay for Quadrant. Secondly, the I/O scores are weighted very heavily... the score nearly tripled because an I/O test completed a little quicker. There's a significant problem with that.
Click to expand...
Click to collapse
Well realistically speaking even without the I/O exploit the none pro version never had any meaning to it..why?
Well it measures:
CPU
Memory
I/O
2d
3d
Now lets look at it from perspective..lets say we have a score of 3000 and 99% of it comes from 3d..the phone would still be laggy because even though it may have a good gpu it won't have the other things to back it up.
Quadrant weighs everything evenly...it exists for the sole purpose of getting people to buy the pro version..
So its not Quadrant that is flawed but the people who actually think a large number means everything is automatically super fast...
So in most cases yes, the I/O increase would be almost useless as nothing would utilize it to the point where you would need to utilize such high I/O..unless maybe you plan to run a database server on your phone? lol
But like any benchmark we have to not only look at the number but understand what the number means..similar to how on a linpack gets cheated on with use of a VFP since it measures with floating points...
Heck I can write a program called gTen Score which does an md5 sum of every file on the device and the closer the device is to mine the higher score it would get..would that make the device with the highest score the best? lol..benchmarks are benchmarks...we have to understand them and look at the details to how the numbers got there..In my opinion if its not Quadrant Pro people are wasting their time with the benchmark...
gTen said:
Well realistically speaking even without the I/O exploit the none pro version never had any meaning to it..why?
Well it measures:
CPU
Memory
I/O
2d
3d
Now lets look at it from perspective..lets say we have a score of 3000 and 99% of it comes from 3d..the phone would still be laggy because even though it may have a good gpu it won't have the other things to back it up.
Quadrant weighs everything evenly...it exists for the sole purpose of getting people to buy the pro version..
So its not Quadrant that is flawed but the people who actually think a large number means everything is automatically super fast...
So in most cases yes, the I/O increase would be almost useless as nothing would utilize it to the point where you would need to utilize such high I/O..unless maybe you plan to run a database server on your phone? lol
But like any benchmark we have to not only look at the number but understand what the number means..similar to how on a linpack gets cheated on with use of a VFP since it measures with floating points...
Heck I can write a program called gTen Score which does an md5 sum of every file on the device and the closer the device is to mine the higher score it would get..would that make the device with the highest score the best? lol..benchmarks are benchmarks...we have to understand them and look at the details to how the numbers got there..In my opinion if its not Quadrant Pro people are wasting their time with the benchmark...
Click to expand...
Click to collapse
You make some very good points, but Quadrant IS flawed in that it should be able to test performance in a manner that is more consistent with real-world performance. A benchmark tool that provides poor results for a different configuration of hardware or software (in this case, Samsung's RFS file system) when performance is in on par or even superior to a higher-scoring configuration is a benchmark tool with serious flaws.
Electrofreak said:
You make some very good points, but Quadrant IS flawed in that it should be able to test performance in a manner that is more consistent with real-world performance. A benchmark tool that provides poor results for a different configuration of hardware or software (in this case, Samsung's RFS file system) when performance is in on par or even superior to a higher-scoring configuration is a benchmark tool with serious flaws.
Click to expand...
Click to collapse
The flaw of quadrant is grouping up all of those categories into 1...usually as you said things are weighted into a benchmark and each benchmark measures things like 3d performance, i/o performance an etc would each be a separate benchmark..Quadrant will never do that because their goal is for you to buy the pro version...it was never intended to be used for anything other then a proof of concept for selling the pro version...
Since people are not gonna buy Quadrant Pro..the best way is to get them to use a different benchmark...I hear some people touting GLBenchmark is it any better?
gTen said:
The flaw of quadrant is grouping up all of those categories into 1...usually as you said things are weighted into a benchmark and each benchmark measures things like 3d performance, i/o performance an etc would each be a separate benchmark..Quadrant will never do that because their goal is for you to buy the pro version...it was never intended to be used for anything other then a proof of concept for selling the pro version...
Since people are not gonna buy Quadrant Pro..the best way is to get them to use a different benchmark...I hear some people touting GLBenchmark is it any better?
Click to expand...
Click to collapse
I've been following GLBenchmark and it seems pretty accurate as far as I can tell. A buddy of mine outed the HTC Glacier (Mytouch 4G) when he found that a developer had erroneously left scores posted on GLBenchmark.com. We could tell it was a next-gen Snapdragon by the scores, and that it wasn't running an Adreno 200. Became big news soon thereafter.
Electrofreak said:
I've been following GLBenchmark and it seems pretty accurate as far as I can tell. A buddy of mine outed the HTC Glacier (Mytouch 4G) when he found that a developer had erroneously left scores posted on GLBenchmark.com. We could tell it was a next-gen Snapdragon by the scores... became big news soon thereafter.
Click to expand...
Click to collapse
Well then the best bet is to contact rom developers and ask if they could include GLBenchmark instead of quadrant and explain the benefits of having a specified benchmark over a general unweighed one..
and yeah I saw th HTC Glacier thing..
Electrofreak said:
This was the whole point of my post... I think you might have missed that. I would also recommend this article as well: http://briefmobile.com/cyanogen-demonstrates-quadrants-flaws
Click to expand...
Click to collapse
You're right, I originally DID miss your point. My bad... I thought you were just another fanboi raving about the awesomeness of Quadrant. I should have read more carefully
But in essence, you and I were saying the same thing all along: Quadrant is bunk and can easily be manipulated by a tmpfs tweak
As for my other point about lagfixes offering real-world performance boosts in other phones, I was saying "yes" in a general way as often times they do, not "yes" specifically regarding Dameon's tweak
biff6789 said:
You're right, I originally DID miss your point. My bad... I thought you were just another fanboi raving about the awesomeness of Quadrant. I should have read more carefully
But in essence, you and I were saying the same thing all along: Quadrant is bunk and can easily be manipulated by a tmpfs tweak
As for my other point about lagfixes offering real-world performance boosts in other phones, I was saying "yes" in a general way as often times they do, not "yes" specifically regarding Dameon's tweak
Click to expand...
Click to collapse
Yeah, and from what I understand (forgive me, I am still a relative newcomer here) the other Galaxy S phones mount their file directory a little differently than the Epic does, negating the need for a real lag fix. However, the Captivate lag-fix, for example, does indeed provide actual performance improvements. Perhaps this misunderstanding is what is causing people to believe that the Epic needs to have some sort of lag fix too.
Electrofreak said:
Yeah, and from what I understand (forgive me, I am still a relative newcomer here) the other Galaxy S phones mount their file directory a little differently than the Epic does, negating the need for a real lag fix. However, the Captivate lag-fix, for example, does indeed provide actual performance improvements. Perhaps this misunderstanding is what is causing people to believe that the Epic needs to have some sort of lag fix too.
Click to expand...
Click to collapse
You're exactly right. Dameon himself stated that the other Galaxy phones need a lagfix while the Epic does not (read the last line of the following post):
http://forum.xda-developers.com/showpost.php?p=9440522&postcount=1174
quadrant has been known to very inaccurate. I forget who it was but someone tweaking their evo was able to get like 6000 on quadrant awhile ago. After that it has been tricked many times into giving big numbers with no actual improvements over the actual phone itself.
The purpose of this post is to inform people of all the major benchmarks, pros/cons or each and what each of them do.
Quadrant Standard/Advanced - This benchmark is made for phones using the Snapdragon proccessor such as the g2/droids/evo etc. Please do not rely on this benchmark for the Epic as the Epic has a Hummingbird.
SmartBench - This benchmark was made to show fair scores for the actual speed of the phone. The developer of this app works very hard to work out "cheats" and bugs. This benchmark seperates the speed in games from the speed in doing other, non-game, tasks.
Linpack - This benchmark, like Quadrant, is mainly for Snapdragon. It shows floating point speed that does not really determine real life performance. Phones like the Evo and G2 get scores way higher than the Epic even though the proccessor on the Epic is faster.
0xBench - This benchmark is a combonation of different benchmarks. The purpose of this benchmark is to find out how fast your phone is for many different things.
Total Benchmark - This benchmark is a FULL benchmark. It not only measures speed but it measures many other things such as accelerometer, display, multitouch, etc.
An3dBench - This benchmark does a wide variety of tests to determine 3d speed
SQLight - This benchmark measures the time it takes to complete a series of non-gaming/non-media tasks.
SetCpu - If you didn't know, SetCpu has 3 different benchmarks included in it. These benchmarks show proccessor speed improvements from your overclock/kernal or rom change.
Nenamark - A GPU benchmark
Neocore - Another GPU benchmark. Made by Qualcomm to showcase their GPU. Since our GPU isnt Qualcomm, you cant fully trust it, but it seems pretty legit.
Did I miss any? Please send me a PM telling me. I know there are alot more but I think I got all of the major ones.
It would be great if this could be stickied.
Reserved for later use
A few others I've got on my phone:
Benchmark Pi: I don't recommend, since all it really does is test one function of the CPU, hardly a comprehensive benchmark.
GLBenchmark: Pretty decent benchmark, all things considered. Lots of options.
NenaMark1: Primarily a GPU benchmark, provides FPS (frames per second) score.
Neocore: Same as the above, mostly tests GPU performance. Created by Qualcomm to showcase their Adreno GPU, so take the FPS scores with a grain of salt.
Another I've heard of, BaseMark. AFAIK its currently only available to software developers. Looking to get my hands on it... looks to be pretty full-featured.
I still recommend SmartBench2010 over anything else right now, mostly because it seems to be accurate and the developer has shown himself to be very dedicated.
Also note AnTuTu. It's comprehensive, and gives a very good breakdown of what goes into its scores.
Added Nenamark and Neocore
Sent from my SPH-D700 using XDA Premium App
GLbenchmarks
Its one of the best. I personally only really care about benchmarks that Anandtech uses since they give the most detailed and comprehensive reviews and testing of new phones and devices. I would check out there site to make sure you got what they use. I know for a fact that GLbenchmarks is one of them. Its a graphical benchmark that tests the GPU. There are also a benchmark to test browser performance, java and i think one other one that they use.
You can get all these on the market?
No, some of these you have to get outside of the market. GLbenchmarks for example is downloaded from their website.
This is just a thread so we can post our quadrant scores.
Highest score will be edited in to this first post.
also a question if you can answer. Why can the nexus s clocked at 1.544ghz score 5000+ on quadrant but we struggle to hit 3000 clocked at 2ghz?
link to video to prove it http://www.youtube.com/watch?v=cAOvEmeriFs
Not to burst your bubble or anything but Quadrant scores have been proven to be useless.
how?
Which would be the best benchmark app to use? Everyone says the same thing about quadrant :S
The I/O testing is flawed and the results don't represent real life.Phones with low Quadrant score can feel smoother than phones with higher Quadrant score eg. Xperia Play vs Galaxy S.
Random.Guy said:
The I/O testing is flawed and the results don't represent real life.Phones with low Quadrant score can feel smoother than phones with higher Quadrant score eg. Xperia Play vs Galaxy S.
Click to expand...
Click to collapse
True
I always thought that the quatrent scores reflected how fast you could MAKE the phone go at optimal conditions... Why can't people just syandardize what you do before you test the phone and leave it at that?
sent from my Experia Play
Random.Guy said:
The I/O testing is flawed and the results don't represent real life.Phones with low Quadrant score can feel smoother than phones with higher Quadrant score eg. Xperia Play vs Galaxy S.
Click to expand...
Click to collapse
This is SO true. When I had my Evo 3d I ran SenselessROM and it felt to be the fastest rom available, but obtained the lowest scores. Didn't make sense at all.