Database Users

UPX - Compression app to free up space

I found this thread on aximsite, a forum I still visit from when I had a Dell Axim and it looked interesting. I didn't find the app in a search here, so I thought I'd post it for anyone who might be interested.
UPX is apparently an app that lets you compress .exe and .dll files without really impacting perfomance in any noticeable way (per the postings in the thread.) I haven't tried it yet, but might give it a shot. It seems the app is not resident on the pocket PC, but needs have files moved to the pc, compressed, then moved back. It's a little bit of work, but it only needs to be done once, so after the set up, it sounds like you're good to go.
Anyway, just thought I'd pass it on as an FYI for anyone who might be interested.
Oh yeah, and it's free too.

Finally UPX is available for WinCE. You should have posted in "General" in the first place.
This program makes your programs even start faster (less to read from usually slow SD card, storage, ...).

For a non-technical person, I am a bit confused... Reading thru the thread, it seems that more ram are required to run the compressed apps.
What exactly do we gain if the apps are stored on a storage card where storage space is not really the issue? Need help.

Wow, thank God. UPX is old skool genius.
Can someone give it a go on the enormous Skype executable? I'm not near my phone for a while and can't check it.
V

vijay, I tested it on the big MobileNavigator.exe which is around 3,7 MB originally. UPX'd only 1,1 MB or so. And it runs! Even Voice Command (300KB down to 100KB). Since I have it installed in Storage it starts faster (Storage is soo slow).

Thank goodness. This is a God send, when reversing PPC binaries you can see that they're full of so much slack.
I had a look at UPX quite a while ago - I use it to compress all my PC VB and C++ stuff, but was disappointed to find it didn't work on PPC. I'm now.
Decompressing presumably incurrs some overhead, but it'll help us out when we've got limited storage.
V

vijay555 said:
Decompressing presumably incurrs some overhead, but it'll help us out when we've got limited storage.
Click to expand...
Click to collapse
I can't agree with that. You can read on the authors page, that
http://upx.sf.net/ said:
Your executables suffer no memory overhead or other drawbacks because of in-place decompression.
Click to expand...
Click to collapse
Although decompression takes some time it is still faster, because it is read into RAM a few times faster than it normally would. This saves more time than it takes.

Good point Chatty. My concern was CPU overhead on slower processors (OMAP), but as you say, since you're using RAM or faster storage, it'll probably compensate.
V

i tried it with opera and tomtom which are both on my main memory
results:
original opera startup 14 sec
upx opera startup 9 sec
original tomtom startup 15 sec
upx tomtom startup 14 sec
i also tried it with tcpmp which is on my SD card
original tcpmp startup 3 sec
upx tcpmp startup 2,5 sec
so it seems to speed things up a little
HOWEVER... in opera the the '-' buttons dont work anymore.. they do work when you use them through the touch screen though..
and the text ('Stop'/'Go to' and 'Menu') is aligned to the left of the screen instead of the normal at 1/3rd and 2/3rd of the screen.
--------------------------------------------
[edit] extra information
original opera 4,86mb
upx opera 1,63mb
original tomtom 2,62mb
upx tomtom 1,67mb
original tcpmp 1,21mb
upx tcpmp 1,21mb
not only a speed gain but it also saves room

OPERA
after a few tests i found out the opera.dll is responsible for the movement of the ´-´ buttons and the not functioning with the keyboard
the opera.exe, xmlparse.dll and zip.dll have nothing to do with it..
with the upx´ed above three and the standard opera.dll startup takes 12 sec so its still some faster then all original files
however the original opera.dll is 4,74mb and the upx opera.dll is 1,57mb, so it doesnt save much room :!:
im just glad ill receive a 4gb SD card today which will give me some extra breathing room compared to my 512mb

Re: OPERA
Bartjan said:
however the original opera.dll is 4,74mb and the upx opera.dll is 1,57mb, so it doesnt save much room :!:
Click to expand...
Click to collapse
1/3! What do/did you expect? I find this rather great!

i meant:
when you use the original opera.dll so the ´-´ buttons still work right, you dont save much room compared to the all original opera :roll:

I went through last night and UPX'd every DLL and EXE that I could, including WisBar (and Desktop), Resco, Opera, PocketInformant and other stuff.
PocketInformant won't start if one of the DLLs is UPX'd, but I don't remember which one. The rest of them and all the EXEs are fine.
Opera - as above. Screw the soft keys, 3MB is a lot of slack to be rid of.
Resco was interesting. The Today plugin messed up at one point, and the Explorer went completely scatty with fonts. Even after uncompressing all the Resco files again the Encrypt/Decrypt context menu items were blank but functioning.
WisBar (compressed) can now live in the 10MB Ext_ROM along with Opera and Resco.
All in all it is something to use carefully, but can save a LOT of space.

Hmm.. so far the responses from the trial seems good. I'll give it a try too later this weekend
--- UPDATE (12 May 06) ---
My results, using UPX -9 (e.g. max compression)
All programs are working so far, but performance are negligible (e.g. less than 2 seconds improvements)
from \Windows
PITools.dll 1.18M -> 504K (save 700K)
from \Program Files
Mobibook.exe : 1.61M -> 619K (save 1.0M)
Tom tom navigator : 1.59 -> 692 (save 900K)
WorldMate.exe : 747K -> 207K (540K)
So, that is a saving of 3M of space! Anyway, for those of you that are adventurous, there are various 500K size dll files in \Windows folder, which I doubt can save much (e.g. the effort/space ratio isn't that attractive).
Anyway, I've found the following files on \Windows that seems to be compressable.
ppt.exe (powerpoint)
gwes.exe (???) <-- DO NOT RUN. It will hang your unit.
gdiplus.dll (???)
msxml3.dll (???)
However, I'm not sure if it is a good idea to compress the ppt that is in ROM, and others are unknown programs (e.g. not sure how to test it after compression)
I've also looked around my SD card and found the Tennis.exe game that I have, that is 2.4M in size which were then compressed downed to 2.17M, which is a bit disappointing. I didn't test if it runs on my unit or not.

hanmin said:
Hmm.. so far the responses from the trial seems good. I'll give it a try too later this weekend
Click to expand...
Click to collapse
An pages anywhere with a list of what NOT to compress?

This is an excellent app! I have compressed all of my apps and have not had any problems.
The big ones for me are PlanMaker and TextMaker - both have compressed to a fraction of their original size. Infact I was running a little tight on space I hadn't even installed TextMaker. But now I have compressed both it isn't a problem.
The only strange one so far is ListPro which works fine but loses it's icon.
I haven't tried to compress everything possible, just the apps and may be the main dll.
I have now got about 5mb more storage space than when I started with TextMaker installed.
Excellent!

hanmin said:
Anyway, I've found the following files on \Windows that seems to be compressable.
ppt.exe (powerpoint)
gwes.exe (???) <-- DO NOT RUN. It will hang your unit.
gdiplus.dll (???)
msxml3.dll (???)
However, I'm not sure if it is a good idea to compress the ppt that is in ROM, and others are unknown programs (e.g. not sure how to test it after compression)
Click to expand...
Click to collapse
You're right that compressing something that is in ROM isn't a good idea, that takes up space!
If you have a file in ROM like \windows\PPT.EXE and it is 1MB in size, then compress it to 500KB, copy it back to \windows, you actually USE 500KB of Program Storage rather than saving it.
Your ROM is Read Only, you can't delete files from it or overwrite them.
What actually happens is that you create another file with the same name and the OS ignores the one in ROM.
I do this with ClearStorage.exe.
I created a .TXT file and in it put
Code:
Clear Storage is dangerous.
then renamed it to ClearStorage.exe. I put it in the \windows folder to stop anyone erasing my device by accident(!).
If I need to clear my device I can delete the file and the original ClearStorage.exe is still there.
Try it yourself.
There is no easy way to save space.
If you want to load your apps to ROM and delete the crap ones that come with the OS this will work, but you will have to decode the ROM, edit the hive files, re-encode the ROM and flash it.
This is no small task, but it can be done (I plan on doing it myself actually).

Wow - This is fantastic. In just a short amount of time I have saved several meg of space, and some programs (such as TomTom) run significantly faster.
Superb!
Cheers
Rowan

Amazing. My Big Storage now BIGGER

Some results (on a Wizard)
MS Communicator Mobile - Compressed the exe and one dll down to 33% (1.3mb). The app would not run with one dll (lclang-res96.dll) compressed.
GoodLink - Compressed GoodApp.exe (the biggest file) from 3.5mb to 1.2mb successfully, however by compressing the largest dll (GoodLinkRes_EN.dll) I lost the soft key labels (and probably some other stuff).
TomTom Navigator - 41%
MS VoiceCommand - Compressed from 1.6mb to .6mb
Those were my big files on my device. Using UPX gained me an extra 6mb storage memory - which was a 50% increase (I went from 12 to 18 ). Off to try it now on my Storage Card apps.
Also, .NET files are not currently supported.

Services.exe, gwes.exe and filesys.exe

I've looked around quite a lot for info on these files but i've had little success with it.
When my Wizard boots up, it has about 16MB of RAM left over. In less than 1 hour without any usage, it goes from 16MB to about 8MB and the phone starts crawling.
Using DotFreds Task Manager, I figured out that the files mentioned in the title grow from sizes of around 500 - 1000 KB at startup to about 4MB each. I use Oxios Hibernate as much as most people use refresh on their computers and inspite of that I can't do much to save RAM.
I've included screenshots with info from DotFreds Task Manager.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Both of them have the exact same apps running with the exception of pMSN server which isnt running in the one with 16MB free. Both have Outlook and ActiveSync open in the background.
As you can see with very minimalistic usage, the memory consumtion has gone through the roof despite the fact that I continuously use Oxios Hibernate to free up memory.

One of the thing I don't really like about M$ is that they tend to have everything under their 'module' and there isn't any way of telling what is causing that particular 'module' to behave as such. For example, in WinXp, there are tons of this svchost.exe running at the background and they are allowed to access the net. And certainly, one day viruses will hijack these 'module' and it will be relatively impossible to detect.
Anyway, the filesys.exe behaviour is very interesting. Based on its name, I would presume that it does file system stuff. So, I wonder what could have made it blown up to such a massive size. What do you DO after you had the unit softreset? If you didn't do anything, I would suspect that the massive other applications might be the culprit, eg. anythings that read/write stuff? It would be helpful if you can list out the Today plugins that you have and, you seems to be usign WA2.

Mr. iPhone... thanks for that
Unlike most members here I hardly have memory resident programs installed.
Today Screen Plugins (This is all that I have set for my TODAY SCREEN)
1) BatteryStatus
2) Default Pocket Outlook Messaging Item
3) Today Agenda
Programs that run in the background
1) WA2
2) SPB Pocket Plus
(I have disabled all features of SPB Pocket Plus including the Today Screen Plugin, with the exception of the Buttons Feature and the Pocket IE Enhancements. I mainly keep this installed because I like the variety of button shortcuts it provides)
3) AE Button Plus
4) PaulyA.com Vista Black by JPL4 (Phone Dialer Skin)
5) Oxios Memory
6) MS VC 1.6
7) SOTI Pocket Controller
8) Saman SMS Delivery Reciept Fix
9) Shubaroo.com ContactFix (Not sure if this is a memory resident program)
Misc Programs that are not active until I start them
1) Resco Explorer
2) Total Commander
3) Opera
4) IM Plus 4.23
5) Macromedia Flash Player ActiveX (not sure as to whether this loads everytime a browser open or if it loads at boot)
6) Skype 2.1
7) z2 R2PC
8) SPB Insight
9) smashcasi RemoteAmp
10) TCPMP Core Player
11) Modaco NO DATA
12) Age of Empires
13) WM5torage
14) Bad sector's ID Transfer v1.00

You can't really do much about those files.
gwes.exe = graphics/windows (the graphic component of WinMob)
services.exe = drivers and stuff.
Filesys.exe = file system stuff (obviously!)
These are fundamental files to the OS. You can't normally disable them or kill them without damaging the OS behaviour. However, what they do is normally dictated by apps you run, drivers you start and other normal phone behaviour. I don't normally watch those files, as their activities are best left alone - consider everything else you're running on your phone though.
Wisbar and Voice Command are hefty apps. Try using a safe mode app and disabling all startup apps, and do a trial and error check of which apps use the most resources.
If you want to get your hands dirty, backup your registry and go through your driver registry and disable unnecessary drivers. I don't recommend that, but you know,
.
V

Thanks for that VJ.
It looks like WA2 and Voice command are using less than 300KB and 1000KB repectively. Tha't quite alright for me. The fact that the services.exe file takes up such a massive chunk of memory is unsettling. Isnt there anyway to have it unload modules and components that are not in use?

Not really. You could, but it's better to stop them from loading to begin with.
If you've got a task manager that will show you module usage (or access to remote tools in Visual Studio/EVC) then find out which modules are a bit elephantine and try to hack through the registry and disable them from loading. It's a very dangerous solution, so backup first.
V

I'm not at guru-level like some, but I have had similar problems in the past, and suggest checking some basics before elephant hunting!!
My experience is that filesys and gwes are the two system modules that absorb the effects of bad memory leaks... IOW, they themselves are not the problem, just the recipients of something else screwing up, which will usually be evidenced in dramatically increased size.
So, my first instinct would be that SOMETHING you have installed is playing badly.
My initial reaction/question is: (this is obvious, but... ) did you change anything recently? New ROM, new app versions, new app settings (e.g., change of OMAP Clock in BatteryStatus), registry tweaks, etc. ?
I use a very similar set of apps. Some thoughts:
* The first thing that jumps at me is Today Agenda, as I know that the recent versions have some instabilities with recent cooked ROMs.
* The second would be to check the interactions between SPB Pocket Plus and WA2 (as task manager) and AE Button Plus (for button assignment), as Pocket Plus doesn;t always play nicely with others, and also has some cooked ROM issues.
* The third would be SPB Insight, which, if running in the background, might be messing things up.
Good luck!

Thanks for that rpodos.
I dunno if it could be any of the things you've mentioned... then again, you never know.
I'm still using the stock rom from iMate
1) SPB Pocket Plus is completely disabled except for the PIE enhancements and the Buttons feature. Hence no conflict between task managers of WA2 and Pocketplus
2) Very rarely use SPB Insight... infact both the screenshots above were taken without an SPB Insight usage.
3) AE Button Plus and SPB Buttons.... hmm... I use the SPB Button Shortcuts in AE Button Plus.
4) Today Agenda... I'm using the latest version, but I only installed it like 2 days ago, and i've been having this same memory problem much before that.

After trying a lot of things that didnt really pan out, I was well on my way to a hard reset when I decided to find a few softwares to replace the SPB functionalities.
1) I got PIE Plus to replace SPB PIE Enhancements
2) Got MonoCube's SafeMode 2
3) PHM's PHM Keys for some of the direct system commands that could be assigned to Buttons to replace the SPB Pocket Plus Button Enhancements.
After I got all this, and was about to Hard Reset, I was just curious to see if the memory situation would imporve marginally by un-installing a few softwares especially Pocket Plus. I went into safe mode and removed the following
1) SPB Pocket Plus
2) PaulyA.com Vista Black by JPL4 (Phone Dialer Skin)
3) SPB Insight
4) Total Commander and
5) z2 R2PC
I restarted and what do you know... no more memory leak. My services.exe file hasn't gone beyond 138 KB.
Filesys.exe and GWES.exe are still at 1468kb and 1202kb, but I guess GWES is understandable since I am using WA2.
Other than that everything else is fine. I installed PIE Plus and it's awesome. I can't believe how incredibly fast it is and how many customizing options it has. I can't believe I used PIE without it all these days. It can do almost everything except give u access to sites like orkut. I guess you can have Opera around for that, but other than some occasional sites like that which could pose a problem, this one really takes the cake. Really fast... without OCing!!!
Definitely recommend that everybody should try it out atleast once.
Just search for "PIEPlus" on www.pocketgear.com
The latest version is 2.2
Anyways, just thought I'd let all of you know that everything is AOK.
In my opinion the memory leak was prolly between SPB Pocket Plus and the Vista Dialer Skin... that thing (Vista Dialer) really drained everything from the day I installed it. Just didn't realise how much till I saw my RAM get depleted by the minute till the wizard was down on its knees at 3MB free ram!!!!! Even opening my Windows folder used to take a millennium till I got rid of the Vista Dialer. Now it opens up in about 5 seconds as opposed to almost 8-12 seconds before.
Oh and lastly... I got between 17.5 MB to 19MB free RAM now with the same setup as before minus the few things I removed as mentioned above.

I think you can keep Total Commander, it is a crime not to have it
anyway, I'm having (running ones, i guess)
phoneAlarm
rlToday
dashBoard
Pocket Breeze
PaulyA.com skin
Has the same amount of free memory as you. I wonder what in my stuff that takes as much memory as the WA2 (pocket breeze? dashboard?). And i would have thought, AE Button Plus and MS VC ought to take more memory. Need to do some investigation this weekend.

Hey buddy, hows it goin?
LOL... yeah... Since I thought I was heading to a hard reset, I just pulled out a few things I wasn't using of late and that included Total Commander, not that it really contributed to the memory hole.
I had Pocket Plus installed from the day I last Hard Reseted and that was on the 15th of Dec 2006. Everything was actually fine. Somewhere mid January, I installed WA2 and even then things were quite alright. I had disabled all of Pocket Plus's features that interfered with WA2 such as Close Button. Somewhere around the same time, I installed the Vista Dialer and soon after that things started going down hill. I guess the memory hole started up because of some sort of incompatability somewhere between these 3 apps. Now that i've gotten rid of Pocket Plus and the Vista Dialer, it all seems fine. I think SPB Insight might have something to do with it as well since it has a service which loads during startup.

Sorry about the posts which are one after the other. Since what I'm mentioning here is a little different from the above, I thought it would be better to put it in a new post.
Just tried uninstalling some of the Today Items DLL's. Though I don't have anything on my Today Screen except for
1) Battery Status
2) Messaging
3) Today Agenda
it seems that the other items which you have the option of selecting for the Today Screen also load their DLL's whether you have enabled them or not.
I had Skype and a whole bunch of other stuff like Pocket MSN etc. I backed up the original settings in the registry and used the Today Plugin Un-Installer to unload the DLL's and remove the entries.
Stuff I've gotten rid off
1) Skype
2) Device Lock (don't really use it)
3) Wireless
4) Tasks
5) Calendar
6) Pocket MSN
After doing this I've gotten about another 500kb fre I think. So yeah.... I'm almost at 20MB free RAM now.

in order to release memory you can use htcAddicts (a quick search would be enough to find it)
it has a function to schedule "clean RAM" daily, every 3 or 6 hours, which might help
all the best

Size matters - how to make png's, exe's, tsk's and cabs smaller

Hello, this is one of the first posts, I don't know if you have any interest on this, but here it goes:
fact 1: EXE's and DLL's may be compressed
fact 2: PNG's may be compressed without any quality loss
based on that, I will explain what I do to make smaller CAB's and do not waste so many space when you have those installed.
tools used:
UPX (http://upx.sf.net)
pngout (http://advsys.net/ken/util/pngout.exe)
msceinf (http://www.codeppc.com/telechargements/msceinf/msceinf.htm)
cabwiz (from Microsoft)
let's put this together with a simple example with HTC Audio Manager CAB file:
size before: 1.135.294
size after: 327.204
(yes, 347% reduction, and all works well)
1. used msceinf to decompile the cab, and decompress it to a directory
2. used command "upx *.exe *.dll" on the decompressed directory
3. used command "for %i in (*.png) do pngout "%i" /kp" on the same directory
4. recreated the cab using "cabwiz Audio Manager.inf /compress"
that's it, until now I could regain a lot of space, of course that using exe compression makes programs a bit slower, but I believe there are more advantages than disavantages.
On Themes you can also achieve better compression using the same technique, decompiling the file, then using pngout on them and recreating the cab's back.
some results on a theme file:
Htc_New_Default.tsk original: 106.828
Htc_New_Default.tsk optimized: 44.835
I also saw some problems on some icon packages: sometimes the authors compile them with thumbs.db on it, resulting in complete waste of space.
Regards.

interesting. I never thought about it this way.
Optimized rom chefs, start your engines!!!!
In other news, dig the title.

It should be pointed out that one of the main negatives to upx'ing files is that they take up more memory. Example..a 100k program upx'd to 50k takes up 150k when ran and not all of the ram is released. This is typically why everyone doesn't just go crazy with compressing everything in site. If you need the space and are willing to sacrifice some memory then upx it..otherwise you'd be better off leaving it be. Also..if you are going to use upx or one of the other utils make sure you are using the most recent version to get best performance.

Yes, that's why I wrote that last sentence regarding EXE compression, some figures for CommManager that occupies 508KB as a process, it was tested by reading the in use memory when CommManager was running, and after I forced it's close (to check real memory usage, since process memory reads the same):
using UPX EXE: 33.77-32.89 = 0,88MB
original EXE: 33.67-32.89 = 0,78MB
so it's about 100KB difference for this EXE (since I don't usually run a lot of programs at the same time, I don't care about this), but for PNG's that are used more and more on pocket pc programs, it really makes a difference.

I've updated this with how to optimize Themes as well...

BullGates said:
I've updated this with how to optimize Themes as well...
Click to expand...
Click to collapse
You know there is a upx --ultra-brute switch that basically tries everything to get the smallest size?

Yes thanks. But as you pointed well, it's better to use EXE and DLL compression evaluating your needs. I've rebuilt many cab's concentrating on the graphics compression and I'm quite happy with the results. Smaller CAB's means smaller install times (not very noticed, but it's a fact) less need for storage - if you build your setups based on extended rom storage, this is quite handy. Also you have better performance since less bytes are involved on the loading (so better loading times).

Actually, I was looking to do exactly the same!
I had already found upx and was going to combine this with infocabxp and my own simple parser for the *.000/_setup.xml files, until I found the free msceinf tool - and about an hour later I find your post.
I just wish I found your post first, it would have saved me a lot of time
It is probably best to apply upx compression on exe's and dll's of standalone applications that you start manually or for short time use, and not all background processes as well, but I see there are already some thoughts on that subject

Thanks
How to do this
. used command "for %i in (*.png) do pngout "%i" /kp" on the same directory
Would u please explain more?
Regards

my 3 cents..
there's no reason to upx small files, lol...
40 kb of space may be equal of 40 kb mem less, if dll is persistent/resident(works all the time) - also not every dll/exe may be upxed - it may be broken after upx(i.e. isilo).
.net apps CANNOT be upxed for now.
DO NOT upx today plugins' dlls! they may lose stability/ you may have loading problems etc + mem loss, of course.
you can upx cpl's too.
always crunch at maximal pack setting!
do not pack exe icons, and relocs.
do not try to make xip modules from upxed files!
gfx resources like bmps may became smaller after pallette decrease(it is quite useful while reshacking - i.e. commgr may be smaller = faster launch times etc) regardless of saving method(24,32bit).
almost 100% of pngs, that are used in software/system etc usually loses its size after reedit = faster software loading/ working/less mem used(dialpad res's, home plug, etc) - usually palette decreasing works in same way, as with bmps - smaller file size(yeah, i know, that png is saved as 8bpp, but..sometimes palette may be 4 bit i.e.).

sygic cant calculate routes above 250 km

hello fellow chefs
i have a problem that whas brought to my attention by a user of my blackstone rom
he uses sygic as navigation and he wanted to calculate a route from netherlands to france and it cant finish calculating as it says it runs out of memory:s
when i tap the taskmaanger it mostly says arround 60% so i gues there is enough also when i look at storage in taskmanager it says
total 191.05 mb
in use 93 mb
free 98.05 mb (these valeus are after a soft reset with 49%)
there is more free then in use so i guess there is enough
also he pointed me to a program where you could test ram wich isnt working on my rom but is working on multiple other roms he has used:S
i also tried it myself by downloading sygic and i came to the same conclusion:S
also the program tells me the same when i want to do a ram check
when i start the program it says at least 21.25 mb free space is needed on the storage device
when i look at taskmanger at the storage section it tells me i have 255 mb free:S
you guys can downlaod the program here: http://rapidshare.com/files/334876190/pocketmechanic.2.99.283.CBD.zip
its a free program so no warezlink
when you instal the program and open it you have the option card benchmark
there you can choose over wich memory you want to perform it
when i say ram it tells me 15% of my ram is used and
302.25mb, 84.68% free
by my calculation 84% of 302 is way more then then the 21 mb the cecker talks about:-o
hope some one can help me solve this strange ram problem as i think the calculation fails of the strange ram thing
edit: gonna put the memorymap from my buildlog here maybe you can see something there
as it says by ram 43 free:-o still its more then the 21 the ram chacker talks about:s
Code:
Memory Map...
SLOT 0: 0x02000000 - 0x018b0000 (END: 0x00060000, 0 MODULES)
0x02000000 - 0x01fc0000 - ROM 0
0x01f80000 - 0x018b0000 - ROM 1
SLOT 1: 0x04000000 - 0x02020000 (END: 0x02020000, 211 MODULES)
SLOT 60: 0x7a000000 - 0x78e40000 (END: 0x78020000, 52 MODULES)
SLOT 61: 0x7c000000 - 0x7a022000 (END: 0x7a020000, 239 MODULES)
RAM IMAGE: 0x80000000 - 0x803e6560
RAM: 0x803e7000 - 0x80475000 - Used for kernel modules
0x80475000 - 0x83000000 - 43 MB free

bumping my 1 post question and got another one
can someone share the htc in call recoreder package including certificates?
thanks in advance

It's probably running out of virtual memory in the slot used by sygic.

Farmer Ted said:
It's probably running out of virtual memory in the slot used by sygic.
Click to expand...
Click to collapse
and can this be changed cause he says he only has the problem with the rom i make:S

How big do you have the datacache (and others in boot.rgu) set? Maybe that's a problem, if it's too big. You should run virtualmemory.exe while the app is running to see if the problem is virtual memory management.

every setting i have is found here on xda so i gues everybody should have the problem then as all of them are in hd2 general section
also the bootrgu i have changed one thing there wich is
[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\FLASHDRV\FATFS]
"DataCacheSize"=dword:00001000 ;4096 sectors(4096*2048=8MB)
normally this whas at 4 mb but found it in a pdf from da_g i believe that you could set it to 8 but on the default valeu and the 8 mb valeu does the same
so i dont thing its this setting thats interferring:-o
will try that program you mentioned
anything special i need to look at there
edit: any chance you can point in the direction where to download that program as i cant seem to find it only source code:S
edit2: never mind after some more googling i found you already posted it somewhere thanks

maybe bit of a noob question but where does the program puts the snapshot:-o
i made two but i cant seem to find them stored anywhere:-o

You need to take a screen-cap; it doesn't save the snapshots for you.

Farmer Ted said:
You need to take a screen-cap; it doesn't save the snapshots for you.
Click to expand...
Click to collapse
lol
i saw snapshot so i thought it whas to make a picture
here is a foto of the phone in idle after a day of use
i first did the snapshot as im not sure what it does
can you see something strange on the picture?

So, you don't have sygic running in that one? Slot 15 is pretty porky; what process is that? You need to scroll left and right to see the processes for each slot. You may need to run vol d-pad to do that on the HD, though. Slots 9 and 10 have low-lying dll's (or some other resource). I suspect that what happens is that sygic loads a resource that is even lower, and maybe it crunches into slot 2 (filesys.exe) or the process in 15. The other possibility is that sygic creates really large heaps when it's doing big calculations (>250 km), and then the memory grows up in that slot to the level where those lower lying dll's are in slots 9 and 10. Even though those dll's (if that's what they are, and not some type of static resource) may not load into the sygic slot, the memory manager still accounts for them in all of the VM slots, so they will limit how big the heaps can grow in any process.
If I were you, I'd run sygic and do a VM snapshot. Also, get devhealth.exe (search on it), run it, and see what those dll's are in slots 9 and 10. They may be the issue. Mainly, see if their shared or static (like a .plg file in tcpmp). Of course, I could be totallly wrong about this, lol.

Farmer Ted said:
So, you don't have sygic running in that one? Slot 15 is pretty porky; what process is that? You need to scroll left and right to see the processes for each slot. You may need to run vol d-pad to do that on the HD, though. Slots 9 and 10 have low-lying dll's (or some other resource). I suspect that what happens is that sygic loads a resource that is even lower, and maybe it crunches into slot 2 (filesys.exe) or the process in 15. The other possibility is that sygic creates really large heaps when it's doing big calculations (>250 km), and then the memory grows up in that slot to the level where those lower lying dll's are in slots 9 and 10. Even though those dll's (if that's what they are, and not some type of static resource) may not load into the sygic slot, the memory manager still accounts for them in all of the VM slots, so they will limit how big the heaps can grow in any process.
If I were you, I'd run sygic and do a VM snapshot. Also, get devhealth.exe (search on it), run it, and see what those dll's are in slots 9 and 10. They may be the issue. Mainly, see if their shared or static (like a .plg file in tcpmp). Of course, I could be totallly wrong about this, lol.
Click to expand...
Click to collapse
muchias gracias for youre help farmer ted
i opend up that virtual memory again and changed to different things
then i opend sygic and took a snapshot again
as you can see its going into slot 14 and every time i took a snapshot the bar went up showing more green(green means in use?)
i just looked up that slot without sygic and it tells me its the slot of tmail.exe
dont think tmail does much in that slot or its a big thing
also on the end i have all those empty slots cant it be reasignd to a new slot so it has a full bar(see the screenshot with sygic calculating a route)
will also try the program you mentioned when i get back from doing my things today as i have some appointments
edit: see you also asked what slot 15 is when i look it up it tells me its manilla.exe and im running sense2.5 from leo 3.04 so it could be heavy

Well, that's it, then. It sounds like sygic has a massive memory leak, if the heap size keeps growing. And this just confirms what I've known since the first day I got my Fuze: Manila sucks, lol.
I'm not sure why the slot 9 and 10 dll's are loaded so low. What processes are those? Maybe something in startup is causing it, and can be fixed.

Farmer Ted said:
Well, that's it, then. It sounds like sygic has a massive memory leak, if the heap size keeps growing. And this just confirms what I've known since the first day I got my Fuze: Manila sucks, lol.
I'm not sure why the slot 9 and 10 dll's are loaded so low. What processes are those? Maybe something in startup is causing it, and can be fixed.
Click to expand...
Click to collapse
thank god inever uused sygic then as it seems a stupid deep going problem
as for manilla sucks im not sure about it
the hd is my first htc device and i think the experience is nice(better then all samsungs i had
for the slots i will write them all down maybe thats easier then asking what is this and what is that(check code box below)
further when i look at windows/startup i have three things there
htc startup, pkg, and poutlook
the htcstartup also has a speaker infront of it with a x behind the speaker(hope you get what i mean)
Code:
slot1: rom dlls
slot2: filesys.exe
slot3: cprog.exe
slot4: backportrait.exe
slot5: device.exe
slot6: gwes.exe(this is a colorfull rom so i can imagine its a little high:p)
slot7: shell32.exe
slot8: virtualmemory.exe
slot9: fexplore.exe
slot10: services.exe
slot11: connmgr.exe
slot12: repllog.exe
slot13: empty
slot14: tmail.exe(sygic will be loaded in this one)
slot15: manilla.exe
slot16: jblenddaemon.exe(have read this can turned off from startup to save a little ram what do you think about that?)
slot17: commmanager.exe
slot18: empty
slot19: empty
slot20: sapsettings
the rest are all empty

I would say remove jblenddaemon.exe; it must be a java app? If you don't use java, there's no reason to have it running. You might sapsettings.exe from startup as well, if you're not using it at all. I removed it a long time ago, but I don't use bluetooth much, and when I do, it works fine without sapsettings. I think I read that it's for connecting to you car or something for hands free.
It's weird that you have the lower lying dll's (if I'm seeing what I think I'm seeing in slots 9 and 10). They definitely seem to be the wall that driver.exe is running into. I also don't know what's up with fexplore.exe. It's using a lot of VM. Is that the extended file explorer? Last time I used it, it would eat a lot of ram if you highlight pics and it showed the previews.

Farmer Ted said:
I would say remove jblenddaemon.exe; it must be a java app? If you don't use java, there's no reason to have it running. You might sapsettings.exe from startup as well, if you're not using it at all. I removed it a long time ago, but I don't use bluetooth much, and when I do, it works fine without sapsettings. I think I read that it's for connecting to you car or something for hands free.
It's weird that you have the lower lying dll's (if I'm seeing what I think I'm seeing in slots 9 and 10). They definitely seem to be the wall that driver.exe is running into. I also don't know what's up with fexplore.exe. It's using a lot of VM. Is that the extended file explorer? Last time I used it, it would eat a lot of ram if you highlight pics and it showed the previews.
Click to expand...
Click to collapse
thanks for the quick overview
the jblenddeamon is the java app
i personally never used it but there are some people using my rom who do use it:-o
guess it will be activated when they start up jblend
im gonna build a rom with jblend and the sapsettings taken out of the bootlauncher and gonna try it again to see what happens
also the file explorer is the normal fexplorer from the 21911 sys
it looks to me the verry same as every other file explorer i have seen on other roms so no preview of pictures you think i need a different fexlorer.exe?
and if so can you recommend one

I was just wondering if you were using the extended file explorer (2.06, I think; by Houming). It's pretty cool, but I personally use total commander and almost never use file explorer, so I don't use the extended on anymore, either. I was never able to make a package with it that worked, so I just did a cab install during customization. The cab is kind of weird, and I can't say I understand how it works. It actually loads up two file explorers (should have remembered that, and known yours wasn't the extended one). I attached the cab for 2.06-I packed this one, and it's compressed and doesn't have the uninstall crap loaded into it.
Sapsettings just doesn't seem to do anything. It doesn't use much memory, but you get quicker boot times when you take stuff like that out of the queue. If anyone really needs it to run, you can just give them a shortcut. You can always just remover the registry keys from your own device after a flash.

Farmer Ted said:
I was just wondering if you were using the extended file explorer (2.06, I think; by Houming). It's pretty cool, but I personally use total commander and almost never use file explorer, so I don't use the extended on anymore, either. I was never able to make a package with it that worked, so I just did a cab install during customization. The cab is kind of weird, and I can't say I understand how it works. It actually loads up two file explorers (should have remembered that, and known yours wasn't the extended one). I attached the cab for 2.06-I packed this one, and it's compressed and doesn't have the uninstall crap loaded into it.
Sapsettings just doesn't seem to do anything. It doesn't use much memory, but you get quicker boot times when you take stuff like that out of the queue. If anyone really needs it to run, you can just give them a shortcut. You can always just remover the registry keys from your own device after a flash.
Click to expand...
Click to collapse
just a quick update as im in a hurry
need to go away to arrange some things for my house in 15 min and i still need to shower:-o
i made a rom without those two thins and damn the rom has bacome faster
also the virtualmemory looks a lot emptier:-o
will post screenshot later today when im finished arranging things
also tried sygic and it managed to calculate alot further but still missing 200 km of the route i want it to be calculated:-o
do you know the differnece between poutlook and tmail.exe?
maybe i can take one of those two out

I'm not really sure what the difference is, to be honest. I don't run either at startup. I don't text much and almost never email, but regardless, you can still do it w/o any issues. I guess maybe outlook loads quicker with one or the other running. I run 11 processes at startup:
nk.exe
filesys.exe
device.exe
gwes.exe
services.exe
shell32.exe
connmgr.exe
cprog.exe
AEBPlus.exe (AE Button)
SpeedBmonitor.exe (Speed Booster; sort of helps, doesn't use much memory)
XHook.exe (XTask task manager)
I also have a shortcut that runs at startup to expand the file dialogue and another to activate resco keyboard, but those processes only run a few seconds each.
That's just 11 processes; the first 6 are mandatory, the next 2 are needed for most things (although you can kill them for maybe a little better performance gaming or something like that-you definitely don't need cprog on an airplane), and the latter 3 are just apps that I run/use all the time. I don't miss any of the other processes, and with fewer processes going, you get faster boot times. My fuze usually boots in 60-65 seconds, which is pretty nice.

well used the rom for a day now as i flashed it yesterday evening before i went to bed
the strangest thing is jblend has activated itself:-o
wonder what triggerd it to get activated as i havent opend java:S
i only used messenegr a little today but its windows its own messenger so i dont think its java related:S
you have any idea what could have opend the jblend?
also the order of what is where has changed but i dont think that does matter as i think all slots are equal of size or have they different sizes?
i will also post a screenshot of the new virtual memory but with jblend in it and sygic gets loaded in that one when i start it up:S
im totally starting to hate this program
you think the screenshots looks better now on how everything is managed?
also i read somewhere if i format my memorycard with bigger clusters it will become faster
what do you think about that?
could this speed up route calculating as the maps or on my sd card
also i will try to remove some more from the startup and hopefully this will give me better sygic calculating things
again thansk or helping em with this probblem as i think i cant solve this alone
Code:
slot1: rom dlls
slot2: filesys.exe
slot3: backportrait.exe
slot4: virtualmemory.exe
slot5: device.exe
slot6: gwes.exe
slot7: shell32
slot8: fexplore.exe
slot9: manila.exe
slot10: services.exe
slot11: connmgr.exe
slot12: cprog.exe
slot13: poutlook.exe
slot14: tmail.exe
slot15: commmanager.exe
slot16: jblendeamon.exe
the rest are all empty

I would check the jblend package and make sure there's not a reg key that's starting it up in the .reg file. The order of processes in the slots depends on the order that the processes are booted up. You can get some processes in an early slot just because there's some transient processes initiated during bootup that only run briefly, then close, which re-opens the slot for the next process in line. If you kill cprog, then start another app, it will go into cprog's slot, so then if you restart cprog, it will be in slot 13 or something like that.
Larger cluster sizes definitely speed up an sd card; I have a 16 g card, which normally had 32 kb clusters. I lowered it once to 4 kb clusters, which freed up maybe 100 mb of memory, but scanning the card became nearly impossible. Normally, if I run scandisk (I do it daily), it takes 5 min. But it would take 40 min or so with the smaller cluster size. Also, benchmarking showed that writing to the card was a lot slower (reading too, I think).
You still seem to have some low lying dll's in slots 9 and 10, which is weird.

[Info] Boot Time per Paging Pool size depending on imgfs compression

I want to share with you some findings on how the size of the paging pool influences the usability of a WM device.
Setup:
Device: HTC Tornado (Smartphone, original WM5, 64MB RAM, 64MB ROM, TI OMAP 850, DiskOnChip from M-Systems)
ROM: my WM6 build (based on Nitrogen's WM6 - see my signature)
ROM Kitchen: OS Builder 1.48
Paging Pool changer: PPSmartchanger (adapted for use without ULDR partition)
Method:
Create builds with OSB selecting the relevant imgfs compression options:
lzx and xpr
with and without having the code sections of modules uncompressed
Change Paging Pool with PPSmartchanger (works only for devices with M-Systems DiskOnChip - so old stuff only - changed with pdocwrite)
retrieve boot-time via devhealth call after device has rebooted to selected standard (for all tests) - 2 repeats have proven enough as there is very little deviation between reboots.
I have taken the boot time because this is easily repeatable and devhealth delivers standardized results.
Results:
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
lzx delivers the smallest ROM builds but also takes longest time to boot
xpr delivers reasonable sized ROM builds and is the fastest
lzx with uncompressed code sections of modules has large ROMs (larger than xpr) and takes longer time to boot
xpr with uncompressed code sections of modules has giant ROMs (had to cut out all optional parts not impacting the procedure) and is not faster than xpr for all modules
Discussion of the results:
You may ask why that setup first? To learn about the paging pool, look also here.
The Paging Pool serves as a dedicated (limited) area of RAM where the current executed code sections of the loaded modules reside. During boot a lot of modules are loaded. While the boot process continues, the demanded use of the paging pool will intermittently exceed its size and pages (parts of the code sections) are discarded to make room for new code to be loaded to the paging pool. As the discarded pages can easily be re-read from their original location on the imgfs - this is no loss of information.
The time it takes to read a module from imgfs, decompress it (lzx, xpr or no compression) and load it to the paging pool for execution determines the total of the boot time. So if you have an unlimited paging pool or you can afford to switch off the demand paging - then your boot time is limited only by read + decompress.
If now the paging pool is smaller than the maximum sequentially loaded modules need it to be, then the discarding of pages in the paging pool will eventually require a re-load of pages for active modules from the imgfs again. The probability for this to happen will increase with a smaller paging pool, so the boot-times will rise with a smaller paging pool. This is what you see for all 4 setups.
Why is lzx imgfs slower than xpr? lzx files are smaller than xpr on imgfs, so they should be read faster from imgfs! True, but obviously the overhead of the decompression to the paging pool is taking more time over all here for my device. So lzx takes more CPU than xpr for decompression.
Why is xpr + uncompressed code sections slower than if the whole imgfs is xpr? The saved overhead of decompressing the code sections should give an advantage here! So here you see that there needs to be balance of imgfs-read performance and CPU power to decompress. For this device it seems that uncompressed larger code sections are no advantage as they take longer to read from imgfs.
Your device's results may vary, but it seems that you only need to measure for one paging pool setup to judge the different options of OS Builder.

Nice comparison. Which boot time marker did you look at?

Just the first in the summary:
<Perf Gates>
<Gate Name=Bootup>27813</Gate>
<Gate Name=Memory>33894400</Gate>
<Gate Name=Storage>13049856</Gate>
</Perf Gates>
This is the same as later:
Bootup Time Markers (Milliseconds):
Available RAM on boot = 34242560
Calupd: Exiting = 30226
Initialized appt ssupdate = 29832
Initialized msgcount ssupdate = 29764
SSUpdate: Thread now in message loop = 29231
Calupd: Starting = 28781
SSUpdate: Shell Ready = 28717
Start Menu Icons Fetched = 28687
Appman: Finished Fetching Icons = 28664
Appman: Started Fetching Icons = 27831
Home Painted = 27813
Cprog: Done Booting = 25188
Last Boot Type = 2
I intend to use that setup also to look a little deeper in the devhealth memory report later, to possibly learn about differences that other OSB options are setting to relevant memory allocation of modules - especially dealing with code that can be paged out or not.
From the MSDN/Kernel Blog articles I have read, there are also other (internal) properties of the code that determine if paging may occur or not. Usually SW production tools set the relevant flags of those automatically so you don't have to care yourself.
I am not enough expert to judge here, but from a glance at the devhealth reports I see that e.g. gwes.exe process has no pageable parts (due to OSB setting it like that?) while other processes (e.g. home.exe) do have pageable code.
I was thinking of e.g. setting rilgsm.dll on the list of modules to prevent demand paging (to have the device immediately respond to incoming calls), but I noticed in the report that it already has no pageable (p) pages listed.
To avoid urban legends I wonder if there is a way to determine or even guide which modules/files are candidates for the list of advanced operations in OSB. It could also help to indicate which are clearly not taking any benefit of those.
From what I understand, OSB has a bunch of options that deal with demand paging one or the other way:
Set Kernel Flag 0x00000001 (MSDN: Demand paging is disabled when the first bit of ROMFLAGS is set. When demand paging is disabled, a module is fully loaded into RAM before running. For OEMs that need real-time performance, turning off demand paging eliminates page faults that can cause poor performance.)
Disable demand paging list (need to find out how this is done - dump directory files are still identical to the SYS folder) - I suspect that relevant code section properties are set here. gwes.exe is internally added automatically (see your build log).
imgfs compression options (see the initial compare in the first post) with the related setting to not compress code sections in modules. It may be helpful to balance the ROM size with potential benefit here if there was option to select the modules where the code sections should stay uncompressed. As seen from below comparison it is no benefit for me - but it may for others.
paging pool size (bigger is better - if you can afford it - even set to "unlimited")
In the context of above I still think that wealthy devices with variable use (like smartphones) better profit from PP = 0 than from Kernelflag = 0x00000001 because an "endless" PP still has the potential recovery of paging out when RAM limit is reached while disabled demand paging prevents further loading of modules at all. The latter may be suitable for fixed purpose devices like navigation devices

Nice, I found this useful. I'm looking forward to your future benchmarks.
I'm wondering how much lzx does impact loading applications after the device has booted-up.

Jackos said:
I'm wondering how much lzx does impact loading applications after the device has booted-up.
Click to expand...
Click to collapse
As long as applications just have to start it does not matter much imho. For such cases I even prefer to UPX these to get even more compression than with the imgfs itself. Here just one time read+decompress has to be taken into account.
The key to the boot-time (and related other module loading and discarding pages later from the paging pool) is that this will happen repeatedly and automatically without you able to steer it.
---------- Post added at 11:37 AM ---------- Previous post was at 11:30 AM ----------
tobbbie said:
I am not enough expert to judge here, but from a glance at the devhealth reports I see that e.g. gwes.exe process has no pageable parts (due to OSB setting it like that?) while other processes (e.g. home.exe) do have pageable code.
I was thinking of e.g. setting rilgsm.dll on the list of modules to prevent demand paging (to have the device immediately respond to incoming calls), but I noticed in the report that it already has no pageable (p) pages listed.
Click to expand...
Click to collapse
I looked up the devhealth report of an old build and the gwes.exe is just the same (from first glance). So I suspect that devhealth is just depicting the current state of the device and not which pages could be paged out (or not). On first fast search I have not found resources to clarify on that yet.

You don't need to run devhealth to get the boot-time, just look at HKCU\Performance\millisecondstoidlethread.
I actually run a mortscript that records each time I reset my device, and how long it takes (there's a shortcut to it in the startup folder). This is the script, if you want to try something similar. It's kind of nice; I've had a few times where my startup time increased a lot, and I like being able to tell when it happened.
Code:
MkDir("\Performance")
If(FileExists("\Performance\bootlog.txt"))
writefile("\Performance\bootlog.txt","^NL^",True)
writefile("\Performance\bootlog.txt",RegRead("HKLM","Comm","BootCount"),True)
writefile("\Performance\bootlog.txt"," boots, booted ",True)
writefile("\Performance\bootlog.txt",FormatTime("m:d:Y:H:i:s a"),True)
writefile("\Performance\bootlog.txt",", reboot time ",True)
sleep(30000)
writefile("\Performance\bootlog.txt",RegRead("HKCU","Performance","Millisec to idle thread"),True)
else
writefile("\Performance\bootlog.txt",RegRead("HKLM","Comm","BootCount"))
writefile("\Performance\bootlog.txt"," boots, booted ",True)
writefile("\Performance\bootlog.txt",FormatTime("m:d:Y:H:i:s a"),True)
writefile("\Performance\bootlog.txt",", reboot time ",True)
sleep(30000)
writefile("\Performance\bootlog.txt",RegRead("HKCU","Performance","Millisec to idle thread"),True)
endif
If(FileExists("\Performance\ramlog.txt"))
writefile("\Performance\ramlog.txt","^NL^",True)
writefile("\Performance\ramlog.txt",RegRead("HKLM","Comm","BootCount"),True)
writefile("\Performance\ramlog.txt"," boots-reset, RAM ",True)
writefile("\Performance\ramlog.txt",FreeMemory(MB),True)
writefile("\Performance\ramlog.txt"," MB, ",True)
writefile("\Performance\ramlog.txt",FormatTime("m:d:Y:H:i:s a"),True)
else
writefile("\Performance\ramlog.txt",RegRead("HKLM","Comm","BootCount"))
writefile("\Performance\ramlog.txt"," boots-reset, RAM ",True)
writefile("\Performance\ramlog.txt",FreeMemory(MB),True)
writefile("\Performance\ramlog.txt"," MB, ",True)
writefile("\Performance\ramlog.txt",FormatTime("m:d:Y:H:i:s a"),True)
endif
This is what the output looks like (the time is milliseconds, so we're looking at~45 seconds per boot):
2 boots, booted 08:11:2011:12:24:24 pm, reboot time
3 boots, booted 09:04:2011:08:04:31 am, reboot time 49245
4 boots, booted 09:04:2011:09:18:14 am, reboot time 46642
5 boots, booted 09:04:2011:13:39:35 pm, reboot time 42994
6 boots, booted 09:12:2011:15:31:33 pm, reboot time 45625
7 boots, booted 09:12:2011:17:30:11 pm, reboot time 45564
Click to expand...
Click to collapse
Edit: Oops, I forgot, only the top part writes the boot log. The second part logs the device free ram whenever I wake it up. I just like to follow that for the hell of it. It's pretty cool to have that log when you go 24 days between soft resets, lol.

...thanks for the scripts - maybe useful as these take very little time to execute. I do a reset every night on my device - so long time RAM "loss" is easily worked around this way. Need to adjust for the keys present in my \performance path though (old WM5 Native Kernel with WM6 OS on top) - have not seen the one you read from there.
I wanted to utilize the devhealth logs later as well to compare more between different settings - this is why I had chosen devhealth to read it for me.
I noticed that some performance gates are linked to the amount of appointments or tasks. These log entries are after "Home painted". So for my compare I simply picked the devhealth selection of "Home painted" as the performance gate for bootup.
This is not the true time until you can use your device, but a good one to compare the influence of a single parameter (the PP size or the imgfs compression).
BTW: I checked if the latest XIP compression from OSB takes an influence on boot time - it does not I have only applied it as recommended to the 2 big ones in my XIP - not sure yet if I want to squeeze the remaining bytes from there.

The following post was initially done in the OSB thread, but I thought it is off topic there, so moved it here:
I read about the use of the paging pool in this blog post but in essence I thought that mostly modules code would be candidates for that memory region. Looking at the output of devhealth there are also other parts (aside modules) marked as "p" which I suspect to bei either "paged out" or "pageable" (have not found a description here).
For me the relevant question are:
Can normally loaded executables (from imgfs as files or elsewhere, e.g. storage card) be utilizing the paging pool? I guess yes - as the OS acts according to the information present in the executable's structure, just like it does for the modules. So if this is the case - what is difference then regarding the use of the paging pool if a certain dll/exe is in "module" format or as a file? Are sections' attributes in a file incarnation set differently regarding the use of paging pool?
Does a potential UPX compression of a file (exe/dll) then impact the utilization of the paging pool and later performance of the exe/dll when paging applies? In a wild speculation and linked to the analogy of imgfs decompression needed when reading LZX compressed code sections parts to be paged in again: would a upx-ed exe/dll need to be re-read and decompressed from storage in case paging of code would apply? I guess no as this would severely impact performance and I have not noticed it for my UPXed (usually BIG) files.
So I wonder if I sacrifice OS memory management flexibility when I UPX compress a large executable, like Opera, that it e.g. needs more RAM and that parts maybe page-able if not UPX-ed and not page-able when UPX-ed. My guess (and hope) is that dll/exe files do not have page-able sections that are compressed. Possibly something to discuss with the UPX-team.
Can I do any tests myself to find out? I could use e.g. Opera and load it one time as UPX-compressed file and another time as normal file. Will the devhealth memory report be suitable to compare these cases? Is the "p" in the devhealth report telling that these pages in memory are "pageable" (so could be paged out if needed) or that they are "paged out" at the time of report?

tobbbie said:
Can I do any tests myself to find out? I could use e.g. Opera and load it one time as UPX-compressed file and another time as normal file. Will the devhealth memory report be suitable to compare these cases? Is the "p" in the devhealth report telling that these pages in memory are "pageable" (so could be paged out if needed) or that they are "paged out" at the time of report?
Click to expand...
Click to collapse
...well I just tried it (using PocketPlayer from Conduits) and the differences are there - and they are as expected:
The normal exe delivers a map that has most memory parts marked either as "-: reserved" or "p-page" and the exe ends at a size of 2 882 088 bytes
The UPX-ed exe delivers a map where executable parts are marked as "E" but ending at a size of 2 775 552 bytes
Strange to notice that the devhealth dump delivers maps that are missing certain memory regions within the sequence of the report. So for the normal report the lines for 2e160000, 2e200000, 2e240000, 2e470000, 2e850000 are missing - compared to the UPX-ed. No clue what these "gaps" mean actually - probably memory that is simply not allocated to the process.
I cannot interpret any detail here, but the obvious difference is that the UPX-ed version seems to allocate the memory in a fixed way (i.e. it cannot by dynamically reclaimed by the OS), while the "normal" version seems to allocate memory in a way that allows "paging" to happen.
UPX:
plus side: no paging happens for the main executable parts - much similar to the OSB module option "exclude from demand paging".
minus side: in total more RAM is required as the whole executable size is loaded (to RAM?).
Normal:
plus side: less RAM is needed as the executable seems to utilize the paging pool
minus side: the paging pool is used, so it should be dimensioned to cope with the parallel use of all normal running applications, not just the modules from ROM.
My speculations are based on the assumption that the "p" indication of the DevHealth report is depicting the utilization of the paging pool. If so then the total number of "p" must not exceed the defined paging pool size. I have no easy way to count these "p" and skip others in the report - so now way to confirm that.
Take-home message: UPX your dll/exe if you want to exclude them from demand paging.
See attachment with the relevant excerpts from the DevHealth reports.

tobbbie, too long way to disable paging for selected exe. You just have to set "unpageable" flag on all sections using apps like LordPE.

ultrashot said:
tobbbie, too long way to disable paging for selected exe. You just have to set "unpageable" flag on all sections using apps like LordPE.
Click to expand...
Click to collapse
Well - for the pro's that only want to have demand paging disabled your suggested way to hack the exe/dll is viable as well. The UPX method has a positive side effect however - the main purpose of UPX is to make the exe/dll smaller - and it does it very well. So you have smaller footprint, faster load time and exclusive RAM with just one activity - and this can be done by anyone, without hacking!
The real downside for those with tight memory (like myself) is that such exclusive memory allocation eats much more precious RAM than if you have the exe/dll allow their code to be paged. So you have to balance the snappy application behavior (UPX) with the number of concurrent running applications demanding memory.
For my part (small RAM) I have taken the following conclusions:
Free RAM is not the goal to optimize for. A small paging pool makes free RAM grow, but it limits the amount of paged-in parts of concurrent running programs.
UPX.ing dll/exe is only useful in case you have plenty of RAM to always keep the whole code in (virtual) memory. So if you have LARGE programs (like my example Pocket Player or Opera) it is advisable NOT to UPX them, despite their storage footprint will be larger. Get these programs on the memory card instead and accept longer loading times. Their RAM footprint will be lower as the OS loads the code to the paging pool (not normal RAM) and it can discard non accessed code parts from the paging pool and reload them from the file again if needed.
Optimize the size of the paging pool to the usually running concurrent programs. This does not mean to minimize the paging pool so that you can still somehow "use" your device, but find a balance for the case that all your concurrent applications are loaded. Mind that any UPX-ed programs (or such marked to be excluded from demand paging) do not count here on the paging pool as their code is put elsewhere! So make sure first that your programs are loaded from a non UPX-format file.
Sigh - all my efforts to get the smallest ROM footprint were done in vain as the price paid is simply the amount of RAM used when such exe/dll are loaded. I need to re-think many optimizations and do a better balancing of ROM size with RAM use.
It is really worth to note that all normal loaded applications have their CODE part loaded to the paging pool, while their other demand for memory (data, heap) is put to other virtual memory. The OS demand paging utilizes this limited amount of RAM (real RAM, not just virtual memory) in an optimal way for the concurrent running programs. So while these execute (and the instruction pointer passes along the memory pages reading the code for execution) the pages have to be in RAM. The overall minimal RAM use (without paging activity) is thus defined by the active code-traces along the memory pages only. All code that is NOT accessed when the programs run can be on discarded pages (so they occupy no RAM).
Now for a mysterious part (to me): The allocations of memory are done for "virtual memory". As you know, each process has 32MB of contiguous virtual memory in its assigned slot - but how does that map to used RAM? So how can I tell how much of precious real RAM is utilized for a certain process? I guess the RAM use for the paging pool is easy understood and also visible in the devhealth report (the "p" - if my assumptions are true), but what about RAM outside the paging pool? Is it just that any part marked in the devhealth report (except the "-" for reserved, as I would count this for discarded code pages) is mapping virtual memory to real RAM 1:1 - so the advantage for the application is just the contiguous address space which in reality may be put on RAM pages that are located anywhere in real RAM address space. From other OS you may remember that there are "paging files" which could also be utilized for mapping virtual memory - but this is not present in WinCE - so is it just 1:1 VM allocation to RAM allocation?
I remember reading that modules loaded from several processes are not loaded several times in the relevant process slot but just referenced from all the processes to their slot1 - so what about the applications data in their slot?
I would really like to learn more about the DevHealth report meaning for the various symbols used when drawing the virtual memory maps. Any hints for me to MSDN or other sources?

I remember reading that modules loaded from several processes are not loaded several times in the relevant process slot but just referenced from all the processes to their slot1 - so what about the applications data in their slot?
Click to expand...
Click to collapse
All module sections except r/w occupy only one space in VM (higher slots on 6.5 kernel, slots 1/0 for older kernels). R/W section is copied to every process using this dll (if that r/w section isn't shareable), but, unfortunately, every process NOT using it will still have this memory reserved.
btw, if you want to read some more interesting reading, here is a nice link

ultrashot said:
All module sections except r/w occupy only one space in VM (higher slots on 6.5 kernel, slots 1/0 for older kernels). R/W section is copied to every process using this dll (if that r/w section isn't shareable), but, unfortunately, every process NOT using it will still have this memory reserved.
Click to expand...
Click to collapse
...well but reserved pages eat no real RAM. It just keeps the process from allocating contiguous memory where the reserved pages are located in VM. As the dlls go top-down and the process bottom up - they have 32MB of VM in the middle. A very nice tool to see this visualized is here: http://www.codeproject.com/KB/windows/VirtualMemory.aspx The "source code" link at the top of the article also contains compiled versions - otherwise I could not have used it.
ultrashot said:
btw, if you want to read some more interesting reading, here is a nice link
Click to expand...
Click to collapse
...oops - looks like a source rip of an old wiki of 2007 - quite hard to get something from it. I found however what I needed on DevHealth here:
https://evnet.svn.codeplex.com/svn/C9Classic/WebSite/Wiki/WikiBases/CEDeveloper/BSPVMDevHealth.wiki

tobbbie said:
...well but reserved pages eat no real RAM.
Click to expand...
Click to collapse
exactly.
10 char

ultrashot said:
exactly.
10 char
Click to expand...
Click to collapse
instead of 4k page size, or?

Ultrashot, if you feel bored sometime you could further give insight to the VM use of a running device's memory snapshot with your devhealthanalyzer tool. The following could as well be discussed in the linked thread of yours if you prefer that. For me it is still linked to the paging pool mainly - so I have it here for a start.
I guess the sum of all "p" give the currently committed pages from the paging pool and so you can see the current percentage of paging pool utilization (as you know the total size from the header of the report. Usually this should be 100% (or close) but if you have huge paging pools it may be less.
I noticed that the devhealth output already reports on the pages utilized per module and process - even giving totals for the system. The use for a 5MB paging pool (1285 pages) is 1267 pages, so 98.6% for a sample I took.
Now the problem is to put that utilization in ratio to the potential demand. You could tell this by comparing the total code size of all loaded exe/dll with the given paging pool how the ratio is between these two.
Evaluating even per loaded exe/dll the ratio of "total code / loaded code to paging pool" could tell how much RAM you would loose/gain if you decide for excluding that from demand paging.
Furthermore the ratio of code (utilizing the paging pool) and other data (stack, heap) eating remaining memory may give advice on the recommended ratio of paging-pool to RAM for a given devhealth snapshot. I think that on average this ratio should be a guide for dimensioning the paging pool - like "reserve 20% or your available RAM for the paging pool".
I don't know which pages from the report should be put in relation to estimate on the above. I had guessed the ratio of "- reserved" to "p" could do it, but the reserved pages are also listed outside the areas which I would attribute to potential paging pool use.
Would not the following be a way to guide a potential finding?
I can grow the paging pool as long as no other memory demand would be limited for my use-case. I know that some applications (video playback, web browsing) have dynamic needs for RAM, but that would be visible in the snapshot taken with devhealth. For an honest report you must not use any memory cleanup tools before taking the snapshot, obviously.
Growing the paging pool beyond the size that all loaded code needs makes no sense and is wasting RAM for a pool that is not exploited.
Shrinking the paging pool to sizes where even the OS (and fancy UI for newer devices) fight for allocated RAM in the paging pool on their own behalf is the lower limit and makes no sense either.
My fault was to assume that you should go as small as possible - but what does it help to have a small paging pool which makes your device slow due to a lot of OS demand paging activity if you still have RAM available that your loaded processes do not utilize for heap and data?!
There are several means to manipulate memory use (mark R/W sections as shared) and avoid the paging pool use (kernel flags, paging pool size = "0", exclude from demand paging - via section flags or simply UPX). I think that only a tool based evaluation of the devhealth output allows to discuss the consequences. Side-by-side compare of different devhealth reports are hard to get insight from - at least for non-pros like myself.
Your module report already gives something but I wonder what conclusion to take from what you list there (module - in use - size - pageable[Y,N,Partly]). Not that much detail as I would need to answer the questions raised further up.