Database Users

Quadrant benchmark for Android on HD2 compared to SGS (What's in a score)

Look at this (from 1:44 on):
It's a quadrant benchmark run on a android port on the HD2. Graphics are really bad, but in the end it has approximately the same score as the benchmarking score of the Galaxy with the original firmware. I mean what is in a score? If I look at the beginning of the movie, the UI is very slow and not as responsive as the Galaxy
(BTW i got 55.7 FPS with the neocore benchmark on JM2)
This is not to say that I don't have deep respect for what the HD2-android development team is doing. Really amazing job. I just can't wait to get my HD2 back from repair.

appelflap said:
Look at this (from 1:44 on):
It's a quadrant benchmark run on a android port on the HD2. Graphics are really bad, but in the end it has approximately the same score as the benchmarking score of the Galaxy with the original firmware. I mean what is in a score? If I look at the beginning of the movie, the UI is very slow and not as responsive as the Galaxy
(BTW i got 55.7 FPS with the neocore benchmark on JM2)
This is not to say that I don't have deep respect for what the HD2-android development team is doing. Really amazing job. I just can't wait to get my HD2 back from repair.
Click to expand...
Click to collapse
Quadrant scores have been criticized for their non-descript breakdowns, at least on their free suite. Also, the fact that they chose the weighting of the scores, so should they chose 2D is equal to 3D weight, I don't know their formula (and for all I know, they give equal weighting to all or they give equal weighting to all test where the CPU has 12 tests and the 3D graphics has 4), but the fact that we, as users don't have access to their formula on their website is a bit unnerving.
Add to that the fact that many reviews and videos rely on it so heavily leaves users a bit misinformed. In reality, and thorough review should definitely run a custom test suite to give individual scores to:
CPU
Memory
I/O
2D graphics
3D graphics
That way users can compare what's important to them. The Galaxy S suffers from terrible I/O and the hacks that have given the fixes typically boost Galaxy scores to nearly double their rates, and it's majorly attributed to improving a bunk I/O score.

Totally agree. In addition, it would be really nice to know which benchmarked factors are responsible for which functions. For example it is really interesting to see how the hd2 performs before the user is running the tests. When the user is scrolling through the setting menu there is a very noticible lag. Given the fact that the total score is nearly the same as the scrore for the SGS, and thar the graphic score of the hd2 is bad in comparisson to the SGS, I would conclude that graphic performance is very important for the way the ui responds.
Sent from my GT-I9000 using XDA App

appelflap said:
Totally agree. In addition, it would be really nice to know which benchmarked factors are responsible for which functions. For example it is really interesting to see how the hd2 performs before the user is running the tests. When the user is scrolling through the setting menu there is a very noticible lag. Given the fact that the total score is nearly the same as the scrore for the SGS, and thar the graphic score of the hd2 is bad in comparisson to the SGS, I would conclude that graphic performance is very important for the way the ui responds.
Sent from my GT-I9000 using XDA App
Click to expand...
Click to collapse
From what I can tell, the HD2 got a decent score 'cos it was running Froyo. When we get bumped up to an official froyo build with JIT fully optimized, We should be top of the pile.

don't forget, android isn't working 100% on the HD2.
I personally think it's pointless comparing to a not complete port.

woops dbl post

alovell83 said:
Quadrant scores have been criticized for their non-descript breakdowns, at least on their free suite. Also, the fact that they chose the weighting of the scores, so should they chose 2D is equal to 3D weight, I don't know their formula (and for all I know, they give equal weighting to all or they give equal weighting to all test where the CPU has 12 tests and the 3D graphics has 4), but the fact that we, as users don't have access to their formula on their website is a bit unnerving.
Add to that the fact that many reviews and videos rely on it so heavily leaves users a bit misinformed. In reality, and thorough review should definitely run a custom test suite to give individual scores to:
CPU
Memory
I/O
2D graphics
3D graphics
That way users can compare what's important to them. The Galaxy S suffers from terrible I/O and the hacks that have given the fixes typically boost Galaxy scores to nearly double their rates, and it's majorly attributed to improving a bunk I/O score.
Click to expand...
Click to collapse
Even then though, it's possible to write a benchmark which wins constantly for any phone.
In regards to "terrible I/O", that might even be due to a bug in the FAT32 drivers. Yes you can benchmark it, but it wont mean much. The best way is to actually TEST the applications you need, rather than select a phone based on benchmarks. However, you are possibly best off looking at the component specs, because they ignore software bugs.

scrizz said:
don't forget, android isn't working 100% on the HD2.
I personally think it's pointless comparing to a not complete port.
Click to expand...
Click to collapse
But the topic is about "what's in a score". Maybe one can generally say that is pointless to compare devices this way. I think that such benchmark scores are only (a bit) relevant at the two poles of the benchmark score spectrum. Everything in between can be neglected due to the uninformed way sub-scores are evaluated.

You got 55.7 FPS on Neocore as the sgs has vertical sync enabled, the refresh rate on the sgs'es screen is 56 fps and thus you can only go up to 56 fps as the v-sync is on. This proves that the sgs is indeed a much more powerful device that is actually being held back. If you can disable the v-sync then you can get a higher fps score

appelflap said:
But the topic is about "what's in a score". Maybe one can generally say that is pointless to compare devices this way. I think that such benchmark scores are only (a bit) relevant at the two poles of the benchmark score spectrum. Everything in between can be neglected due to the uninformed way sub-scores are evaluated.
Click to expand...
Click to collapse
I just read in a post that the Galaxy S gets a 0 on the 2D score:
"JIT isn't fully enabled in the current froyo versions, and quadrant, frankly, is bull**** (for exmple, 2d acceleration gets the same weight in the final result as 3D. Due to the fact that the SGS doesn't have a dedicated 2D accelerator, quadrant doesn't try to use the cpu- it just gives a round zero in that part)"
I can't confirm this, but that definitely seems like a terrible set-up, seeing as how I'm pretty sure I have games run in 2D, so to say that it can't do it just seems wrong regardless of if the SGS has a dedicated 2D accelerator or not (so if you aren't testing the way it performs in real-world, why are you testing?)
http://forum.xda-developers.com/showthread.php?t=737787&page=3

Qazz~ said:
You got 55.7 FPS on Neocore as the sgs has vertical sync enabled, the refresh rate on the sgs'es screen is 56 fps and thus you can only go up to 56 fps as the v-sync is on. This proves that the sgs is indeed a much more powerful device that is actually being held back. If you can disable the v-sync then you can get a higher fps score
Click to expand...
Click to collapse
It isn't really being held back - the screen can't display more than 56 fps as you say, and it wouldn't really be visible even if it could. Disabling v-sync isn't really that important, we need a benchmark that can actually use the advanced features in the SGS GPU (Neocore just pushes a fairly small amount of polygons with no real extras.) Using current 3D benchmarks to benchmark the SGS is like using quake 1 to benchmark the brand new ATI/nVidia cards.
The benchmark is what is at fault here, not the device

RyanZA said:
It isn't really being held back - the screen can't display more than 56 fps as you say, and it wouldn't really be visible even if it could. Disabling v-sync isn't really that important, we need a benchmark that can actually use the advanced features in the SGS GPU (Neocore just pushes a fairly small amount of polygons with no real extras.) Using current 3D benchmarks to benchmark the SGS is like using quake 1 to benchmark the brand new ATI/nVidia cards.
The benchmark is what is at fault here, not the device
Click to expand...
Click to collapse
I don't want to speak for the other poster, and I agree with your premise, however, it isn't actually solving the issue at hand. Better FPS wouldn't be noticed, however, it would give a better score and, more importantly, indicate it's potential. So, getting 56FPS isn't doing the phone any justice within the score, which is what reviews are using, giving it an artificially low score, and putting it more in line with units that can't compete on higher end games. So, when a site like anand pushes 150FPS on a game, I know that means that their rig is entirely too powerful for the game in question, but it still means something when you compare it to the lower end graphics card that only gets 90...then when they run Crisis you see these results play out more with differences that we can notice with the eye.

I think the HD2 gets that score because, as I can see in the video, the CPU tests run faster compared to my SGS, probably because of Froyo, and I know, from the time I had the Diamond and the HD2, that the internal memory and RAM are very fast. Sadly SGS has a slow internal memory, atleast when used by the phone`s software, when copying from PC is faster than my class 6 microSD. Luckily, we have mimocan`s fix. Hope this will be fixed in future FW`s.

NexusHD2 with-FRG83D V1.7 with hastarin r8.5.1 On my HD2 got 1920 in quadrant,31.5 on neocore, and 37 on linmark.

The lag might be because you are using launcher pro, I use launcher pro and sometimes it makes the the lock lag on my phone but it doesn't happen when I use the default lock also if you have alot of Widgets on your screen it will cause lag also

appelflap said:
Look at this (from 1:44 on):
It's a quadrant benchmark run on a android port on the HD2. Graphics are really bad, but in the end it has approximately the same score as the benchmarking score of the Galaxy with the original firmware. I mean what is in a score? If I look at the beginning of the movie, the UI is very slow and not as responsive as the Galaxy
(BTW i got 55.7 FPS with the neocore benchmark on JM2)
This is not to say that I don't have deep respect for what the HD2-android development team is doing. Really amazing job. I just can't wait to get my HD2 back from repair.
Click to expand...
Click to collapse
same galaxy s scores 6000+ in quadrant with custem roms

The HD 2 is a better fit for quadrent then the sgs as quadrent was made for the snapdragon processor which the hd2 has and the sgs does not. Comparing apples to orenges in an apple juice contest doesn't really prove much. Use real life feel. If you care about the scores a rom can be made to get you over 3000 quad score but is laggy as hell. Don't believe me? Look at my sig

interesting... I was using quadrant to see how a stock xxjvo and gingerreal compared. Surely that would indicate a real speed difference and not just be some kind of "hack" ?

zelendel said:
The HD 2 is a better fit for quadrent then the sgs as quadrent was made for the snapdragon processor which the hd2 has and the sgs does not.
Click to expand...
Click to collapse
That's right.
HD2 uses two android OS :
- Cyanogenmod, that is faster than our samsung os..
- Nexus one's port to HD2, greatly optimized by google...
It's really fast. I upgraded my father's HD2 last month, replacing windows in the NAND with CM7. It really makes a big change, the phone is like brand new ^^

http://forum.xda-developers.com/showthread.php?t=1012556
Quadrant is pretty flawed. And I say that being someone who had a phone (before modifications) that was mid-range in Quadrant (Galaxy S), and having a phone that's right top of the heap (Galaxy S II)

Has anyone been working on a way to uncap our FPS?

I am highly interested in this phone as a gaming device and I know it would be awesome to see this phone truly benchmarked video wise. The droidx beats it because of uncapped FPS even though it has a worse GPU.

don't worry too much about it. 56 is a really good number. once more graphics intensive games come out, our phones will shine at 56 FPS while others will struggle.

I thought it was already determined that the FPS wasn't capped. I know on my phone when running quadrant it does go above 56 but seems to average around 56. Besides I don't see any games out right now that would benefit from high frame rates, except maybe ps1.

speoples20 said:
I thought it was already determined that the FPS wasn't capped. I know on my phone when running quadrant it does go above 56 but seems to average around 56. Besides I don't see any games out right now that would benefit from high frame rates, except maybe ps1.
Click to expand...
Click to collapse
It is not the point of the games but if our fps is uncapped right now, the droidx gets more fps than our phone with a older model GPU. 60 FPS is all you need for games but Im saying, why would I want a SGS for video games with apparently a DroidX gpu is better according to benchmarking.

I think the lower scores vs. the Droid 2 is more an indication of how lousy Quandrant is as a benchmark than any indication of the actual potential of the SGS.

^^
The droid X does NOT have a higher fps. I've tested one, an incredible, rooted/stock droid, nexus,etc....
None of these phones came close to 56.7 fps like the Vibrant.
Not sure where you got that, but it's incorrect.

Mark271 said:
It is not the point of the games but if our fps is uncapped right now, the droidx gets more fps than our phone with a older model GPU. 60 FPS is all you need for games but Im saying, why would I want a SGS for video games with apparently a DroidX gpu is better according to benchmarking.
Click to expand...
Click to collapse
Find anywhere where the Droid x gpu is faster. Necore is like 42 fps vs sgs 55+, nenomark 1 is 44 on sgs and what? 22 on dx?
Your info is wrong, now the dx may jump that high during testing at one point, but average the sgs holds steady framerates much better.

There is a cap though. No matter what I can never go above 56 fps on any app.
Sent from my SGH-T959 using XDA App

smashpunks said:
Find anywhere where the Droid x gpu is faster. Necore is like 42 fps vs sgs 55+, nenomark 1 is 44 on sgs and what? 22 on dx?
Your info is wrong, now the dx may jump that high during testing at one point, but average the sgs holds steady framerates much better.
Click to expand...
Click to collapse
I had seen a video of reviews and was surprised when they said the Droid X got like 59 FPS on quadrant while the SGS got 56

Mark271 said:
I had seen a video of reviews and was surprised when they said the Droid X got like 59 FPS on quadrant while the SGS got 56
Click to expand...
Click to collapse
But that's untrue. Sgs wins in every gpu benchmark.

smashpunks said:
But that's untrue. Sgs wins in every gpu benchmark.
Click to expand...
Click to collapse
Hmm, I wish I could find the video, I am too lazy to look through the 1000 of comparison videos.
But granted, maybe I was mistaken.

^ trust me, I've tested both... whatever you say you saw, again, it is incorrect.

Mark271 said:
Hmm, I wish I could find the video, I am too lazy to look through the 1000 of comparison videos.
But granted, maybe I was mistaken.
Click to expand...
Click to collapse
I've seen probably every dx and sgs gpu benchmark video out. I can 100 % guarantee that the sgs not only beats the dx, but whips it by a good margin. They may have said at point during the test the dx may go over 60 at some points, but its all over the place going from 40 to 60 back down and up and etc. Where as the sgs just chills nicely around 56 although I've seen it jump to 58 its just all around more smooth and steady framerate vs the dx.....
Here's a video where they run the sgs against the dx, n1 and evo. Clearly see in neocore that the sgs eats the others alive and also in real world testing on them. Sgs clearly even beats the dx.
http://www.youtube.com/watch?feature=youtube_gdata_player&v=FRK2stnV3Wg

Haha okay. My bad. I would still like an uncapped to get a proper benchmark.
Sent from my SGH-T959 using XDA App

Mark271 said:
Haha okay. My bad. I would still like an uncapped to get a proper benchmark.
Sent from my SGH-T959 using XDA App
Click to expand...
Click to collapse
Why are you so into mines is bigger than yours?

I have an idea but I don't think it is worth posting...

[GUIDE] How to benchmark properly

This thread is a guide on how to benchmark your phone.
The Problem
Many people are posting screenshots of their quadrant scores with captions like "9th run 6000 " which isn't representative of real use. It also perpetuates the notion that quadrant alone can be used to indicate how well the kernel/governor/scheduler/ROM are performing.
The Solution
If you enjoy running quadrant lots of times, you might as well collect all the data from each run in a spreadsheet. Use separate rows for CPU Memory I/O 2D and 3D.
At the end of each row add a formula like this: =average(a1:j1)
...or select the appropriate range for however many runs you did.
The more runs you do the better representation of the true mean you get.
Additionally, you could also include this formula: =median(a1:j1)
...which orders the results and gives you the middle one or a value in between the middle two depending on whether you did an odd or even number of runs respectively.
After doing more than say, 7 runs, once the mean and median start to converge, you know you have a true mean. (Assuming normally distributed data - sometimes referred to as a 'Bell Curve')
To increase the accuracy, you could split the difference between the mean and the median. Add them and divide by two. Or use: =average(mean,median)
...replacing mean and median with the correct cell reference.
Variability
Some scores vary only a little around the true mean. Some vary a lot more. If you want to work out a percentage for how much they vary, you can use something called the standard deviation.
Get the spreadsheet to work it out for you: =stdev(a1:j1)
One standard deviation around the mean will show you where you'd expect to find 'most' results, about 66% of all the tests you run should be in that range. Two standard deviations will give you 95%. Three is 99.7%.
We'll accept 95% as good enough. Here's how to get the variability percentage:
=1-((mean-2*stdev)/mean)
... replace mean and stdev with the correct cell references.
The "mean-2*stdev" bit is the lower boundary above which we would find 95% of the results. (The upper boundary could be calculated by using a plus instead of a minus, but you would end up with an improper fraction.) We use this as the numerator over the mean, and subtract the result from 1. This gives us some number below 1. Now select that cell, and tell the spreadsheet to format it as a percentage rounded to 2 decimal places.
You now have your variability expressed as a percentage of the mean. The lower this value the more you know the benchmark always produces a reliable result.
If you are going to be comparing kernels or ROMs, try to eliminate all user definable variables, such as choice of governor and scheduler, by making the same selection every time, and turn all radios off (except wifi for certain benchmark apps) turn off sync, location services, etc.
If that sounds like a lot of effort, you can always refer to benchmarks I have already produced: here.
Unreliable Results
The following apps' results vary considerably (more than 5%), and may need several runs to obtain an average:
AnTuTu > Database IO
Quadrant > CPU, Memory, and I/O
Smartbench 2012 > Gaming Index
Vellamo > Aquarium Canvas, Pixel Blender, V8 benchmark

Reserved 1

Reserved 2

Thanks for the info

great guide very useful
thanks

Thank you. I reached my 8 thanks limit for the day, so when I hit the thanks button I get a message "Sorry, you have too much gratitude for XDA servers to cope with."
In just the last few minutes I realised that quadrant is now basically a completely useless tool for benchmarking. Check this link to find out why:
http://forum.xda-developers.com/showpost.php?p=22323688&postcount=194

A510 Quadrant scores average 2650- Now the average is 3720

Update: NOW the tests show 3720 average, so not sure what is going on....
I have done a bunch of tests and the average is 2650: Range as low as 2500 and 2760 was the highest. I thought people were getting about 1000 higher than that?
Been using the newest version of Quadrant and restarted the A510 as well and see no apparent signs of slowness and emulators & media play great. Something is not adding up here. Without doubt the 510 is faster than my Thrive, but that is not saying a whole lot (Tegra 2 is nearing two years old).

rushless said:
Update: NOW the tests show 3720 average, so not sure what is going on....
I have done a bunch of tests and the average is 2650: Range as low as 2500 and 2760 was the highest. I thought people were getting about 1000 higher than that?
Been using the newest version of Quadrant and restarted the A510 as well and see no apparent signs of slowness and emulators & media play great. Something is not adding up here. Without doubt the 510 is faster than my Thrive, but that is not saying a whole lot (Tegra 2 is nearing two years old).
Click to expand...
Click to collapse
I tweak the kernel of the A500 on ICS and the quadrant score is 2500-2600
But don't look to hard on the quadrant benchmark , it's probably your I/O score that is low.

[Q] Benchmark extremely low?

No matter how i try i usually get 12k on AnTuTu
my highest was 14k
Im lower than HoX and in the chart Nexus 4 shows 18k...
Anyone knows why mine is that much lower?
Is this a defect or what?
Some guy says ignore it benchmarks mean nothing but its not like im just a little bit lower...
It is 4-6k difference :S
Isn't that strange considering other people with the same device score 18k?

ruzkay said:
No matter how i try i usually get 12k on AnTuTu
my highest was 14k
Im lower than HoX and in the chart Nexus 4 shows 18k...
Anyone knows why mine is that much lower?
Is this a defect or what?
Some guy says ignore it benchmarks mean nothing but its not like im just a little bit lower...
It is 4-6k difference :S
Isn't that strange considering other people with the same device score 18k?
Click to expand...
Click to collapse
Is your phone laging, running slower other then the benchmarks? Also we need more info like what kernel/rom are you running.

What ROM/kernel are you running, what temperature was your phone at? Is there any apps running in background?
If a freshly installed stock ROM, with only AnTuTu installed, and you score 30% less than other Nexus 4 in benchmarking, then yes, there is a reason to worry about
Edit: for the record, I tried AnTuTu when I just got my N4, and always scored lower than their reference N4.
Sent from my Nexus 4 in a Faraday cage

I tried on these setups:
Stock with stock kernel
Stock with faux kernel
AOKP with included kernel
i just tried again with a fresh install of stock and scored 14,5k
well its still 3,5k difference what can I do about that any idea?
edit: basically what im worried about is that the only scores i see around my score are people with severely underclocked devices (600 MHz, 700 MHz)
seems like im the only one :S

Benchmarks mean nothing. Don't bother looking at them.

zephiK said:
Benchmarks mean nothing. Don't bother looking at them.
Click to expand...
Click to collapse
if they dont mean anything then why do they exist?
I'm not worried about scoring low, I'm worried about scoring considerably lower than the exact same device.
There must somehow be a reason for it I guess?
Meh nvm I'm just sad that mine is possibly the weakest Nexus 4 on the planet get only 18 fps in that lighning 3d test with the 2 characters fighting...
e: please close this thread it was a stupid question to ask I guess I'm sorry

OP:
I am sorry to hear your frustration. Did you check your CPU binning yet? I can't recall how my phone perform on the AnTuTu test, but maybe you can try to measure fps when you play *actual* game?
zephiK said:
Benchmarks mean nothing. Don't bother looking at them.
Click to expand...
Click to collapse
Even Chainfire makes benchmark app. I would love to see how you tell Chainfire that his app means nothing. That would make a really interesting conversation with lots of good stuffs to learn
From my point of view, benchmark does not tell real world performance, however it is a wildly used, totally valid tool to compare *hardware* performance. After all, benchmarking (or hardware profilling, to be precise) is what separate a 1.2GHz CPU from a 1.5GHz CPU of identical architecture, isn't it? I'll take the 1.5GHz one even if it means nothing from your point of view
Sent from my Nexus 4 in a Faraday cage

I always get around mid to high 17k, sometimes 18... bone stock... Your situation is interesting to say the least...

I see the note II with 24k despite the fact it has quad A9s... This is why you should ignore the benchmarks.
Sent from my Nexus 4 using xda app-developers app

Sounds like you may be thermal throttling. As a test, out your phone in a zip lock bag in the freezer for 10 minutes and then repeat the benchmark.

mattb3 said:
Sounds like you may be thermal throttling. As a test, out your phone in a zip lock bag in the freezer for 10 minutes and then repeat the benchmark.
Click to expand...
Click to collapse
wow i knew of thermal throttling but didn't think it kicks in in only 1 benchmark!
yes thats it! i tried the benchmark while putting the phone on a very cool surface (glass desk)
and it scored 17k the 1k difference left is redundant i think^^
KyraOfFire said:
OP:
I am sorry to hear your frustration. Did you check your CPU binning yet? I can't recall how my phone perform on the AnTuTu test, but maybe you can try to measure fps when you play *actual* game?
Click to expand...
Click to collapse
i don't usually do gaming but i tried dead trigger just for testing
it does have some fps drops sometimes but it is totally playable
but wow now i know this device can get EXTREMELY hot, good thing im not a gamer cause i like the feeling of cold class^^
well thanks all guess its normal then!