Database Users

UPX - Compression app to free up space

I found this thread on aximsite, a forum I still visit from when I had a Dell Axim and it looked interesting. I didn't find the app in a search here, so I thought I'd post it for anyone who might be interested.
UPX is apparently an app that lets you compress .exe and .dll files without really impacting perfomance in any noticeable way (per the postings in the thread.) I haven't tried it yet, but might give it a shot. It seems the app is not resident on the pocket PC, but needs have files moved to the pc, compressed, then moved back. It's a little bit of work, but it only needs to be done once, so after the set up, it sounds like you're good to go.
Anyway, just thought I'd pass it on as an FYI for anyone who might be interested.
Oh yeah, and it's free too.

Finally UPX is available for WinCE. You should have posted in "General" in the first place.
This program makes your programs even start faster (less to read from usually slow SD card, storage, ...).

For a non-technical person, I am a bit confused... Reading thru the thread, it seems that more ram are required to run the compressed apps.
What exactly do we gain if the apps are stored on a storage card where storage space is not really the issue? Need help.

Wow, thank God. UPX is old skool genius.
Can someone give it a go on the enormous Skype executable? I'm not near my phone for a while and can't check it.
V

vijay, I tested it on the big MobileNavigator.exe which is around 3,7 MB originally. UPX'd only 1,1 MB or so. And it runs! Even Voice Command (300KB down to 100KB). Since I have it installed in Storage it starts faster (Storage is soo slow).

Thank goodness. This is a God send, when reversing PPC binaries you can see that they're full of so much slack.
I had a look at UPX quite a while ago - I use it to compress all my PC VB and C++ stuff, but was disappointed to find it didn't work on PPC. I'm now.
Decompressing presumably incurrs some overhead, but it'll help us out when we've got limited storage.
V

vijay555 said:
Decompressing presumably incurrs some overhead, but it'll help us out when we've got limited storage.
Click to expand...
Click to collapse
I can't agree with that. You can read on the authors page, that
http://upx.sf.net/ said:
Your executables suffer no memory overhead or other drawbacks because of in-place decompression.
Click to expand...
Click to collapse
Although decompression takes some time it is still faster, because it is read into RAM a few times faster than it normally would. This saves more time than it takes.

Good point Chatty. My concern was CPU overhead on slower processors (OMAP), but as you say, since you're using RAM or faster storage, it'll probably compensate.
V

i tried it with opera and tomtom which are both on my main memory
results:
original opera startup 14 sec
upx opera startup 9 sec
original tomtom startup 15 sec
upx tomtom startup 14 sec
i also tried it with tcpmp which is on my SD card
original tcpmp startup 3 sec
upx tcpmp startup 2,5 sec
so it seems to speed things up a little
HOWEVER... in opera the the '-' buttons dont work anymore.. they do work when you use them through the touch screen though..
and the text ('Stop'/'Go to' and 'Menu') is aligned to the left of the screen instead of the normal at 1/3rd and 2/3rd of the screen.
--------------------------------------------
[edit] extra information
original opera 4,86mb
upx opera 1,63mb
original tomtom 2,62mb
upx tomtom 1,67mb
original tcpmp 1,21mb
upx tcpmp 1,21mb
not only a speed gain but it also saves room

OPERA
after a few tests i found out the opera.dll is responsible for the movement of the ´-´ buttons and the not functioning with the keyboard
the opera.exe, xmlparse.dll and zip.dll have nothing to do with it..
with the upx´ed above three and the standard opera.dll startup takes 12 sec so its still some faster then all original files
however the original opera.dll is 4,74mb and the upx opera.dll is 1,57mb, so it doesnt save much room :!:
im just glad ill receive a 4gb SD card today which will give me some extra breathing room compared to my 512mb

Re: OPERA
Bartjan said:
however the original opera.dll is 4,74mb and the upx opera.dll is 1,57mb, so it doesnt save much room :!:
Click to expand...
Click to collapse
1/3! What do/did you expect? I find this rather great!

i meant:
when you use the original opera.dll so the ´-´ buttons still work right, you dont save much room compared to the all original opera :roll:

I went through last night and UPX'd every DLL and EXE that I could, including WisBar (and Desktop), Resco, Opera, PocketInformant and other stuff.
PocketInformant won't start if one of the DLLs is UPX'd, but I don't remember which one. The rest of them and all the EXEs are fine.
Opera - as above. Screw the soft keys, 3MB is a lot of slack to be rid of.
Resco was interesting. The Today plugin messed up at one point, and the Explorer went completely scatty with fonts. Even after uncompressing all the Resco files again the Encrypt/Decrypt context menu items were blank but functioning.
WisBar (compressed) can now live in the 10MB Ext_ROM along with Opera and Resco.
All in all it is something to use carefully, but can save a LOT of space.

Hmm.. so far the responses from the trial seems good. I'll give it a try too later this weekend
--- UPDATE (12 May 06) ---
My results, using UPX -9 (e.g. max compression)
All programs are working so far, but performance are negligible (e.g. less than 2 seconds improvements)
from \Windows
PITools.dll 1.18M -> 504K (save 700K)
from \Program Files
Mobibook.exe : 1.61M -> 619K (save 1.0M)
Tom tom navigator : 1.59 -> 692 (save 900K)
WorldMate.exe : 747K -> 207K (540K)
So, that is a saving of 3M of space! Anyway, for those of you that are adventurous, there are various 500K size dll files in \Windows folder, which I doubt can save much (e.g. the effort/space ratio isn't that attractive).
Anyway, I've found the following files on \Windows that seems to be compressable.
ppt.exe (powerpoint)
gwes.exe (???) <-- DO NOT RUN. It will hang your unit.
gdiplus.dll (???)
msxml3.dll (???)
However, I'm not sure if it is a good idea to compress the ppt that is in ROM, and others are unknown programs (e.g. not sure how to test it after compression)
I've also looked around my SD card and found the Tennis.exe game that I have, that is 2.4M in size which were then compressed downed to 2.17M, which is a bit disappointing. I didn't test if it runs on my unit or not.

hanmin said:
Hmm.. so far the responses from the trial seems good. I'll give it a try too later this weekend
Click to expand...
Click to collapse
An pages anywhere with a list of what NOT to compress?

This is an excellent app! I have compressed all of my apps and have not had any problems.
The big ones for me are PlanMaker and TextMaker - both have compressed to a fraction of their original size. Infact I was running a little tight on space I hadn't even installed TextMaker. But now I have compressed both it isn't a problem.
The only strange one so far is ListPro which works fine but loses it's icon.
I haven't tried to compress everything possible, just the apps and may be the main dll.
I have now got about 5mb more storage space than when I started with TextMaker installed.
Excellent!

hanmin said:
Anyway, I've found the following files on \Windows that seems to be compressable.
ppt.exe (powerpoint)
gwes.exe (???) <-- DO NOT RUN. It will hang your unit.
gdiplus.dll (???)
msxml3.dll (???)
However, I'm not sure if it is a good idea to compress the ppt that is in ROM, and others are unknown programs (e.g. not sure how to test it after compression)
Click to expand...
Click to collapse
You're right that compressing something that is in ROM isn't a good idea, that takes up space!
If you have a file in ROM like \windows\PPT.EXE and it is 1MB in size, then compress it to 500KB, copy it back to \windows, you actually USE 500KB of Program Storage rather than saving it.
Your ROM is Read Only, you can't delete files from it or overwrite them.
What actually happens is that you create another file with the same name and the OS ignores the one in ROM.
I do this with ClearStorage.exe.
I created a .TXT file and in it put
Code:
Clear Storage is dangerous.
then renamed it to ClearStorage.exe. I put it in the \windows folder to stop anyone erasing my device by accident(!).
If I need to clear my device I can delete the file and the original ClearStorage.exe is still there.
Try it yourself.
There is no easy way to save space.
If you want to load your apps to ROM and delete the crap ones that come with the OS this will work, but you will have to decode the ROM, edit the hive files, re-encode the ROM and flash it.
This is no small task, but it can be done (I plan on doing it myself actually).

Wow - This is fantastic. In just a short amount of time I have saved several meg of space, and some programs (such as TomTom) run significantly faster.
Superb!
Cheers
Rowan

Amazing. My Big Storage now BIGGER

Some results (on a Wizard)
MS Communicator Mobile - Compressed the exe and one dll down to 33% (1.3mb). The app would not run with one dll (lclang-res96.dll) compressed.
GoodLink - Compressed GoodApp.exe (the biggest file) from 3.5mb to 1.2mb successfully, however by compressing the largest dll (GoodLinkRes_EN.dll) I lost the soft key labels (and probably some other stuff).
TomTom Navigator - 41%
MS VoiceCommand - Compressed from 1.6mb to .6mb
Those were my big files on my device. Using UPX gained me an extra 6mb storage memory - which was a 50% increase (I went from 12 to 18 ). Off to try it now on my Storage Card apps.
Also, .NET files are not currently supported.

Size matters - how to make png's, exe's, tsk's and cabs smaller

Hello, this is one of the first posts, I don't know if you have any interest on this, but here it goes:
fact 1: EXE's and DLL's may be compressed
fact 2: PNG's may be compressed without any quality loss
based on that, I will explain what I do to make smaller CAB's and do not waste so many space when you have those installed.
tools used:
UPX (http://upx.sf.net)
pngout (http://advsys.net/ken/util/pngout.exe)
msceinf (http://www.codeppc.com/telechargements/msceinf/msceinf.htm)
cabwiz (from Microsoft)
let's put this together with a simple example with HTC Audio Manager CAB file:
size before: 1.135.294
size after: 327.204
(yes, 347% reduction, and all works well)
1. used msceinf to decompile the cab, and decompress it to a directory
2. used command "upx *.exe *.dll" on the decompressed directory
3. used command "for %i in (*.png) do pngout "%i" /kp" on the same directory
4. recreated the cab using "cabwiz Audio Manager.inf /compress"
that's it, until now I could regain a lot of space, of course that using exe compression makes programs a bit slower, but I believe there are more advantages than disavantages.
On Themes you can also achieve better compression using the same technique, decompiling the file, then using pngout on them and recreating the cab's back.
some results on a theme file:
Htc_New_Default.tsk original: 106.828
Htc_New_Default.tsk optimized: 44.835
I also saw some problems on some icon packages: sometimes the authors compile them with thumbs.db on it, resulting in complete waste of space.
Regards.

interesting. I never thought about it this way.
Optimized rom chefs, start your engines!!!!
In other news, dig the title.

It should be pointed out that one of the main negatives to upx'ing files is that they take up more memory. Example..a 100k program upx'd to 50k takes up 150k when ran and not all of the ram is released. This is typically why everyone doesn't just go crazy with compressing everything in site. If you need the space and are willing to sacrifice some memory then upx it..otherwise you'd be better off leaving it be. Also..if you are going to use upx or one of the other utils make sure you are using the most recent version to get best performance.

Yes, that's why I wrote that last sentence regarding EXE compression, some figures for CommManager that occupies 508KB as a process, it was tested by reading the in use memory when CommManager was running, and after I forced it's close (to check real memory usage, since process memory reads the same):
using UPX EXE: 33.77-32.89 = 0,88MB
original EXE: 33.67-32.89 = 0,78MB
so it's about 100KB difference for this EXE (since I don't usually run a lot of programs at the same time, I don't care about this), but for PNG's that are used more and more on pocket pc programs, it really makes a difference.

I've updated this with how to optimize Themes as well...

BullGates said:
I've updated this with how to optimize Themes as well...
Click to expand...
Click to collapse
You know there is a upx --ultra-brute switch that basically tries everything to get the smallest size?

Yes thanks. But as you pointed well, it's better to use EXE and DLL compression evaluating your needs. I've rebuilt many cab's concentrating on the graphics compression and I'm quite happy with the results. Smaller CAB's means smaller install times (not very noticed, but it's a fact) less need for storage - if you build your setups based on extended rom storage, this is quite handy. Also you have better performance since less bytes are involved on the loading (so better loading times).

Actually, I was looking to do exactly the same!
I had already found upx and was going to combine this with infocabxp and my own simple parser for the *.000/_setup.xml files, until I found the free msceinf tool - and about an hour later I find your post.
I just wish I found your post first, it would have saved me a lot of time
It is probably best to apply upx compression on exe's and dll's of standalone applications that you start manually or for short time use, and not all background processes as well, but I see there are already some thoughts on that subject

Thanks
How to do this
. used command "for %i in (*.png) do pngout "%i" /kp" on the same directory
Would u please explain more?
Regards

my 3 cents..
there's no reason to upx small files, lol...
40 kb of space may be equal of 40 kb mem less, if dll is persistent/resident(works all the time) - also not every dll/exe may be upxed - it may be broken after upx(i.e. isilo).
.net apps CANNOT be upxed for now.
DO NOT upx today plugins' dlls! they may lose stability/ you may have loading problems etc + mem loss, of course.
you can upx cpl's too.
always crunch at maximal pack setting!
do not pack exe icons, and relocs.
do not try to make xip modules from upxed files!
gfx resources like bmps may became smaller after pallette decrease(it is quite useful while reshacking - i.e. commgr may be smaller = faster launch times etc) regardless of saving method(24,32bit).
almost 100% of pngs, that are used in software/system etc usually loses its size after reedit = faster software loading/ working/less mem used(dialpad res's, home plug, etc) - usually palette decreasing works in same way, as with bmps - smaller file size(yeah, i know, that png is saved as 8bpp, but..sometimes palette may be 4 bit i.e.).

Batch upx compress (compress whole folders of exes)

Hi folks,
In the course of making my own ROM (something I admit to being new at) I found it rather tedious to individually compress exes with the upx utility and batches included with HyperCore kitchen. I've made a batch job that will look through all files in an input folder and compress all the exes (so you can just copy your entire SYS and OEM folder out of your kitchen, compress their contents and copy back). Perhaps something to do this already exists, but if so I couldn't find it. I've also included a more recent version of upx with this than was in HyperCore.
Hopefully this will save someone a little time.
To use:
1. Copy/Move folders containing exes (I suggest just copying the whole SYS and OEM folders) into the input folder.
2. Run compress.bat
3. Copy folders in output back to your kitchen.

While UPX is good for compressing PE files (which results in a smaller output file), one should point out, that it's a tradeoff. You get a smaller file, but a higher startup latency, and sometimes a tad larger memory footprint. So UPXing is not good for responsivenes of a device.
UPXing small .exe files may then be beside the point. You'll gain for eg. 5KB but you'll loose a second every time you load the file.
UPXing usually makes sence on executable files larger then 200-400 KB, and smaller then 15MB.
You can google for UPX performance for more information.
Just my 5 cents, and I think it's important for cookers.

NIXin said:
but a higher startup latency
Click to expand...
Click to collapse
mehehehe.
heh. it was not really good text, imo(maybe it is, on e-105..but even omap have VERY fast decrunch speed).
4 cents are ok, in general...
my one cent:
test EVERY upxed dll, EVERY upxed exe, some things will be broken after operation(isilo - i.e.).
very good idea is to pack office pack exes(much space for useful soft while cooking).
.net apps CANNOT be uxped for now.
imo, batch upxer is just NOT good idea...

Good input from both of you, particularly in regard to testing the executables. There is an option to force byte-identical decompression, perhaps it would be a good idea to enable this? I am aware that there is a trade-off in performance with compression, but reading the readme on upx one notes that decompression is very fast. Even assuming our ppcs are 10 times slower than the Pentium 133 decompression would still be 1MB/sec, and most of our exes are much smaller than that.
Just curious if this case where files don't work after compression is a common one (noting I'm not compressing dlls as they are unable to share data when compressed; this is noted in the upx documentation, where they comment it's probably only a good idea to compress things like plugin dlls). Perhaps someone with more experience can comment on this?

caeci11ius said:
Good input from both of you, particularly in regard to testing the executables. There is an option to force byte-identical decompression, perhaps it would be a good idea to enable this? I am aware that there is a trade-off in performance with compression, but reading the readme on upx one notes that decompression is very fast. Even assuming our ppcs are 10 times slower than the Pentium 133 decompression would still be 1MB/sec, and most of our exes are much smaller than that.
Just curious if this case where files don't work after compression is a common one (noting I'm not compressing dlls as they are unable to share data when compressed; this is noted in the upx documentation, where they comment it's probably only a good idea to compress things like plugin dlls). Perhaps someone with more experience can comment on this?
Click to expand...
Click to collapse
While compression is necessary to fit some things in it really ought to be a last resort. I recall the author stating that a 100k program compressed to 50k requires 150k of memory to expand and run and it doesn't free up all of the extra memory needed...so there has to be a balance or you are going to be using up all your memory to save storage....if the app HAS to be installed to phone storage then upx is something to consider but I wouldn't recommend globally upx'ing all your apps on your minisd for instance.

.net apps CANNOT be uxped for now.
Click to expand...
Click to collapse
Because .NET files don't have the standard PE header. It's a requirement for UPX (or any other executable file packer) (PL: pozatym witam )
Mr. famewolf just confirmed what I have said and even added some more. It's perfectly okay for PCs, where you don't notice the difference, but PPC's are just way to slow and the gain is too little (if any). Batch method is not the way to go, unless you wan't to "cripple" your PPC.

UPX and paging, performance effect?

Hi,
I am thinking about the sense behind UPX for mobile devices.
From what I understand about the mechanism in WM5/6, the following should be correct. I post this not as fact, but as my personal thesis on the effects of UPX, so please correct me if I am wrong.
The devices have a small pagepool, and rely on discarding and reloading of code segments.
So, the operating system needs to "forget" parts of a loaded executable if the place in the pagepool is small, and reload these parts if they are to be executed again. With UPX, this could be a real pain as these parts of the file are not stored in the rom in their real-time-executable form, but need some "decompression". So the device is either not able to discard parts of upxed files, resulting in less pagepool-space for other files, or needs to expand them at reloadtime, resulting in a lot of cpu-load as parts of large executables may need to be expanded several times over the execution span of a program.
Has anyone measured the effects of the use of a lot of UPX-ed executables in a device? I would be very interested in the effect upxing has on modules.
Regards,
TG.

[Q] How to protect my asset folder?

Hey I'm building an app with an asset folder full of sound files. I'd like to protect it from decompilation. I understand it's considered impossible but I'd like to use the example of console games like Playstation's. I remeber they pack their files in custom .DAT files that are indexed. The whole files are packed and it can be painful to extract the files without lnowing the way they are packed. I think that is the best way to do.
Is there a tool that can do that for me?

basterd2 said:
Hey I'm building an app with an asset folder full of sound files. I'd like to protect it from decompilation. I understand it's considered impossible but I'd like to use the example of console games like Playstation's. I remeber they pack their files in custom .DAT files that are indexed. The whole files are packed and it can be painful to extract the files without lnowing the way they are packed. I think that is the best way to do.
Is there a tool that can do that for me?
Click to expand...
Click to collapse
this is Dalvik and JIT, so you can't really do what you say... PS is about as far from mobile/java as you can get.... maybe don't use java and use NDK for such things

basterd2 said:
Hey I'm building an app with an asset folder full of sound files. I'd like to protect it from decompilation. I understand it's considered impossible but I'd like to use the example of console games like Playstation's. I remeber they pack their files in custom .DAT files that are indexed. The whole files are packed and it can be painful to extract the files without lnowing the way they are packed. I think that is the best way to do.
Is there a tool that can do that for me?
Click to expand...
Click to collapse
If you download games like asphalt or need for speed (or any big game for that matter), you will notice that they download additional data (expansion files) to the sdcard which has media and other stuff and they are stored in some different extensions (not mp3 or wav). But I have no idea how it is done. You probably will have to write binaries to encode and decode them.

gh0stslayer said:
If you download games like asphalt or need for speed (or any big game for that matter), you will notice that they download additional data (expansion files) to the sdcard which has media and other stuff and they are stored in some different extensions (not mp3 or wav). But I have no idea how it is done. You probably will have to write binaries to encode and decode them.
Click to expand...
Click to collapse
THis is very interesting, I'm checking the android page, they have a tool that does exactly that, it creates an opaque binary blob. The thing is I'm wondering how to use it directly witrhout using their expansion pack system.
http://developer.android.com/tools/help/jobb.html

You can protect your assets while you encrypt them before you put them in the apk. In your app you have to decode them of course.
You don't need a complex encoding system. Maybe you just add 5 to each byte... This may not be 100% sure, but protect from just playing the mp3/wav/ogg file.

EmptinessFiller said:
You can protect your assets while you encrypt them before you put them in the apk. In your app you have to decode them of course.
You don't need a complex encoding system. Maybe you just add 5 to each byte... This may not be 100% sure, but protect from just playing the mp3/wav/ogg file.
Click to expand...
Click to collapse
Can you give me an code example that does that? I've battling the APKexpansion files for hours with no success.