Related
I figured I would start a tutorial thread. If you have some tutorials or instructions you would like to share then post them. Also post links to other helpful posts and threads.
Basic instructions for building an android squashfs system image:
First you need a Linux installation. I use Ubuntu 9.04 and run as the root user.
You need to have squashfs-utils installed. I have version 3.3 installed.
You need the android version of genext2fs. You might want to chmod this to executable before using it.
Starting simple download the ion 1.5 build or any other build that works on vogue as a base.
If the base image your using is a system.img then it's cramfs. You can mount this image and pull from it. Enter android-builds directory and create a tmp directory for mounting the image and a mount point in that tmp dir.
Code:
mkdir tmp
mkdir tmp/cram-sys
mount -o loop system.img tmp/cram-sys
The system.img should be mounted on tmp/cram-sys
Now you have a base system image to grab files from.
If you have a system.sqsh then it is a little easier to do.
Put the system.sqsh in a working directory like base-rom in your android-builds directory and while in the directory from a terminal execute:
Code:
unsquashfs system.sqsh
This will create a directory called squashfs-root.
Then download a build from the dream forums on xda.
Usually they come in update.zip format. Extract the zip to a working directory like android-builds/foo
Inside you should see some directories mainly data and system is what you are after.
Look inside the data folder for any app or app_s directories. Copy those apps to the system/app directory. Leave out any unnecessary apps like wifi tether and so on.
In the system directory is another directory called lib. You need to take out libaudio.so libaudioflinger.so libgps.so. Depending upon the build the vogue specific libraries are either in /lib or /lib/donut in the rootfs. We take these out of system/lib because they cause conflicts. Another way around this is to copy the vogue specific libraries to system/lib.
In the system directory is another directory called etc. You need to replace apns.conf with the one from a working vogue android build.
Depending on the build you are porting you will need to modify system/bin and system/xbin. I usually add any missing files to the new image and base the missing files off of the ones in the android 1.5 ion build.
After this you need to edit your build.prop in the system directory. Look at the ion cupcake 1.5 or a donut build with market fix for reference depending on the build you are porting.
The last step is to put a .build file in the system directory. This depends on the build type.
donut.build
ion.build
rogers.build
hero.build
After you've modified the system directory you need to use genext2fs to create an android specific ext2 image.
for cupcake/donut images
Code:
genext2fs -d ./system -b 80000 -a system.ext2
this will make ~80mb ext2 system image.
for hero/sense ui builds a larger image is required
Code:
genext2fs -d ./system -b 160000 -a system.ext2
This system.ext2 is now bootable and you should be able to copy it to your sdcard and run it.
To turn this into a system.sqsh you need to mount it
Code:
mkdir tmp/system-ext2-mnt
mount -o loop system.ext2 tmp/system-ext2-mnt
Then make a squashfs image of the mounted directory
Code:
mksquashfs tmp/system-ext2-mnt/ system.sqsh
Build scripts/environment from loserskater:
http://forum.xda-developers.com/showthread.php?t=585452
ADB setup and Telenet via adb:
enatefox's tutorial for setting up adb and udev
enatefox's tutorial for telenet via adb
Adding a hosts file to block adds
Download this hosts file and copy it to /etc/hosts this should block most adds while browsing in android. Saving you time and bandwidth.
Using google dns servers on your phone:
(mssmision)
connect to the adb shell or from the terminal with su in android type
setprop ro.kernel.android.ndns 2
setprop net.eth0.dns1 8.8.8.8
setprop net.eth0.dns2 8.8.4.4
Basic Theming:
http://forum.xda-developers.com/showpost.php?p=3175504&postcount=1
Tethering via tetherbot and adb.
Download and install adb on your computer.
Download tetherbot and install it in android.
Connect your phone to your computer
Open tetherbot on your phone and push "Start Socks"
Open a terminal on your computer and type
Code:
adb forward tcp:1080 tcp:1080
Open firefox on your computer and type
Code:
about:config
search in the filter for
Code:
network.proxy.socks_remote_dns
change the value from true to false
Go into your network preferences settings in firefox and select manual proxy configuration
In the SOCKS host field enter localhost and TCP port 1080
Select SOCKS_v5
Click OK
Reload firefox and you should be running off your phones data network.
thank you Zen will give it a blast
oooh yeah runs faster!!!
smarty lol do you still need to keep sd card in slot after flashing?
# mkdir /mnt
# mkdir /mnt/rootfs-sd
# mkdir /mnt/rootfs-nand
# mkdir /mnt/system-sd
# mkdir /mnt/system-nand
# mkdir /mnt/data
# mkdir /mnt/sdcard
# flash_eraseall /dev/mtd/mtd1
Erasing 128 Kibyte @ d80000 -- 100 % complete.
# flash_eraseall /dev/mtd/mtd2
Erasing 128 Kibyte @ 1a20000 -- 29 % complete.
Skipping bad block at 0x01a40000
Erasing 128 Kibyte @ 5980000 -- 100 % complete.
# flash_eraseall /dev/mtd/mtd3
Erasing 128 Kibyte @ 75e0000 -- 92 % complete.
Skipping bad block at 0x07600000
Erasing 128 Kibyte @ 8000000 -- 100 % complete.
are the bad blocks there normal?
ali3nfr3ak said:
# mkdir /mnt
# mkdir /mnt/rootfs-sd
# mkdir /mnt/rootfs-nand
# mkdir /mnt/system-sd
# mkdir /mnt/system-nand
# mkdir /mnt/data
# mkdir /mnt/sdcard
# flash_eraseall /dev/mtd/mtd1
Erasing 128 Kibyte @ d80000 -- 100 % complete.
# flash_eraseall /dev/mtd/mtd2
Erasing 128 Kibyte @ 1a20000 -- 29 % complete.
Skipping bad block at 0x01a40000
Erasing 128 Kibyte @ 5980000 -- 100 % complete.
# flash_eraseall /dev/mtd/mtd3
Erasing 128 Kibyte @ 75e0000 -- 92 % complete.
Skipping bad block at 0x07600000
Erasing 128 Kibyte @ 8000000 -- 100 % complete.
are the bad blocks there normal?
Click to expand...
Click to collapse
yea thats normal
here is what i did to get it working
1. unsquashfs the system.sqsh
2. move apps folder to desktop
3. then mksquashfs system.sqsh
4. in terminal do(make sure no data.img on desktop)
Code:
dd if=/dev/zero of=./desktop/data.img bs=1048576 count=128
mke2fs -F ./desktop/data.img
sudo mkdir /mnt/data
sudo mount -o loop ./desktop/data.img /mnt/data
sudo mv ./desktop/app /mnt/data/app_s
sudo umount /mnt/data
then copy the data.img and system.sqsh to sdcard
and then make
datafstonand
and
systemfstonand
then right after it starts to boot
A N D R I O D
enabled adb bla bla
then do
adb shell stop
then
ln -s /data/app_s /system/app
then
adb shell start
you should be good if you followed this to the T
jamezelle said:
yea thats normal
here is what i did to get it working
1. unsquashfs the system.sqsh
2. move apps folder to desktop
3. then mksquashfs system.sqsh
4. in terminal do(make sure no data.img on desktop)
Code:
dd if=/dev/zero of=./desktop/data.img bs=1048576 count=128
mke2fs -F ./desktop/data.img
sudo mkdir /mnt/data
sudo mount -o loop ./desktop/data.img /mnt/data
sudo mv ./desktop/app /mnt/data/app_s
sudo umount /mnt/data
then copy the data.img and system.sqsh to sdcard
and then make
datafstonand
and
systemfstonand
then right after it starts to boot
A N D R I O D
enabled adb bla bla
then do
adb shell stop
then
ln -s /data/app_s /system/app
then
adb shell start
you should be good if you followed this to the T
Click to expand...
Click to collapse
you should put the symlink in the system folder before you squash it that way it should boot straight through.
ali3nfr3ak said:
smarty lol do you still need to keep sd card in slot after flashing?
Click to expand...
Click to collapse
if you edit init and put cache in /data you should be able to fully run from nand.
or from adb shell
Code:
cd /
umount /cache
mkdir /data/cache
mount /cache /data/cache
pull your sdcard. look ma running from nand.
zenulator said:
you should put the symlink in the system folder before you squash it that way it should boot straight through.
Click to expand...
Click to collapse
one problem i have is i dont think compcache is being enabled, becaulse im getting home reloads every(yes everytime i hit home(hold cam/redbutton))
can i just add the commands from my hero.user.conf to init or init.hero.rc, or will they not be called right?
jamezelle said:
one problem i have is i dont think compcache is being enabled, becaulse im getting home reloads every(yes everytime i hit home(hold cam/redbutton))
can i just add the commands from my hero.user.conf to init or init.hero.rc, or will they not be called right?
Click to expand...
Click to collapse
when i ran the tattoo build from nand compcache was enabled. make sure you have the config files in your android directory on your sdcard. to check if you have swap enabled go into the terminal or adb shell and type
Code:
free
and see how much ram/swap you have.
also i think
Code:
cat /proc/swaps
will show you what swap is being used.
Zen can u tell me what all the files i need on my sdcard to get this going, i tried for a couple of hours last night to get this going no luck tho, and jamezelle helped me out as well still no luck tho, all i had was on my sdcard was android folder with dzos rootfs.img, system-noapps.sqsh, VOGUIMG.nbh as i flashed from sdcard and app_s.tar.gz
[email protected]:~# adb shell
# mkdir /mnt
# mkdir /mnt/rootfs-sd
# mkdir /mnt/rootfs-nand
# mkdir /mnt/system-sd
# mkdir /mnt/system-nand
# mkdir /mnt/data
# mkdir /mnt/sdcard
# flash_eraseall /dev/mtd/mtd1
Erasing 128 Kibyte @ d80000 -- 100 % complete.
# flash_eraseall /dev/mtd/mtd2
Erasing 128 Kibyte @ 1a20000 -- 29 % complete.
Skipping bad block at 0x01a40000
Erasing 128 Kibyte @ 5980000 -- 100 % complete.
# erase_eraseall /dev/mtd/mtd3
/system/bin/sh: erase_eraseall: not found
# flash_eraseall /dev/mtd/mtd3
Erasing 128 Kibyte @ 75e0000 -- 92 % complete.
Skipping bad block at 0x07600000
Erasing 128 Kibyte @ 8000000 -- 100 % complete.
# mount -t vfat /dev/block/mmcblk0p1 /mnt/sdcard/
# losetup /dev/block/loop1 /mnt/sdcard/android/rootfs.img
# mount /dev/block/loop1 /mnt/rootfs-sd
# mount /dev/block/mtd1 /mnt/rootfs-nand/
mount: mounting /dev/block/mtd1 on /mnt/rootfs-nand/ failed: No such file or directory
i cannot get past this part, Do i need more files on sdcard to get going?
ali3nfr3ak said:
Zen can u tell me what all the files i need on my sdcard to get this going, i tried for a couple of hours last night to get this going no luck tho, and jamezelle helped me out as well still no luck tho, all i had was on my sdcard was android folder with dzos rootfs.img, system-noapps.sqsh, VOGUIMG.nbh as i flashed from sdcard and app_s.tar.gz
[email protected]:~# adb shell
# mkdir /mnt
# mkdir /mnt/rootfs-sd
# mkdir /mnt/rootfs-nand
# mkdir /mnt/system-sd
# mkdir /mnt/system-nand
# mkdir /mnt/data
# mkdir /mnt/sdcard
# flash_eraseall /dev/mtd/mtd1
Erasing 128 Kibyte @ d80000 -- 100 % complete.
# flash_eraseall /dev/mtd/mtd2
Erasing 128 Kibyte @ 1a20000 -- 29 % complete.
Skipping bad block at 0x01a40000
Erasing 128 Kibyte @ 5980000 -- 100 % complete.
# erase_eraseall /dev/mtd/mtd3
/system/bin/sh: erase_eraseall: not found
# flash_eraseall /dev/mtd/mtd3
Erasing 128 Kibyte @ 75e0000 -- 92 % complete.
Skipping bad block at 0x07600000
Erasing 128 Kibyte @ 8000000 -- 100 % complete.
# mount -t vfat /dev/block/mmcblk0p1 /mnt/sdcard/
# losetup /dev/block/loop1 /mnt/sdcard/android/rootfs.img
# mount /dev/block/loop1 /mnt/rootfs-sd
# mount /dev/block/mtd1 /mnt/rootfs-nand/
mount: mounting /dev/block/mtd1 on /mnt/rootfs-nand/ failed: No such file or directory
i cannot get past this part, Do i need more files on sdcard to get going?
Click to expand...
Click to collapse
well the last command you typed should be
Code:
mount -t yaffs2 /dev/block/mtdblock1 /mnt/rootfs-nand
also i updated the tutorial to reflect that and also a simplified version along the lines of jamezelle's method of that uses dzo's normal installer so you shouldn't have to type all those commands just use the blank files in the rootfs and make sure you have the rest in /andoroid on your sdcard.
Not sure if you've seen the thread I started on the Kaiser forums. It's really easy to run scripts that I made for building systems.
Check it out
loserskater said:
Not sure if you've seen the thread I started on the Kaiser forums. It's really easy to run scripts that I made for building systems.
Check it out
Click to expand...
Click to collapse
I added a link in the first post. great work.
Installing adb drivers on Windows [Vista,7] x64
NOTE: You don't need to disable driver signing to install! The appropriate .cat files are included in the download.
//Prep
1. Download adb2 posted by dzo. It's on the sourceforge site:
adb2
2. Unpack the zip.
3. In android, go to settings - applications - development and make sure usb debugging is selected.
//Cleanup
4. Open device manager (start - run - devmgmt.msc)
5. Plug in your phone and _watch_ for something new to show up. Might be 'Android Phone.'
6. Confirm that the new device is your phone by unplugging. The device should disappear.
7. Plug back in.
8. Right click on the device and choose 'Uninstall'
9. Check the 'delete this driver from my computer' box.
10. click ok. NOTE: It may take a loong time. Wait.
11. Unplug your phone.
12. If prompted to reboot, do so.
13. Uninstall WMDC (Windows Mobile Device Center)
14. Reboot.
//Install
15. Re-download and reinstall WMDC
16. Plug in your phone.
17. The device driver installation wizard should fail. IF IT INSTALLS THE DEVICE, go back to cleanup.
18. Open device manager.
19. Right click on 'Android Phone' and choose 'update driver.'
20. Click 'Browse my computer for driver software.'
21. Click 'let me pick'
22. Click 'next'
23. Click 'have disk'
24. Click 'browse'
25. Browse to the location where you unzipped adb2.
26. Open the Vista_x64 directory.
27. Choose androidusb.inf
28. Click ok
29. Choose 'HTC, Corporation' as the manufacturer (may not see this)
30. Choose 'My HTC' as the model
31. Click 'next'
32. Click 'yes'
DONE!
Zen, I know we were talking yesterday. Do you think you can outline in detail how to convert an EXT2 or NAND or SD Card based installed system on the vogue into something we can distribute?
I'd really like to share my customized rom with people.
Thanks again for all of your contributions
Ok finally got tattoo booting was having wakeup problems but made system .ext2 then sqsh. But audio still not playing, any ideas? Thanx I did use audio libs from donut rootfs
i thought the libs had to come from an ION build?
Hi all,
I've been trying hard to replicate the 'echo mode' of SyncToy (Microsoft freeware) using rsync. If you want to understand how this mode acts, here is an illustration:
ht***tp://ww***w.laboratoire-microsoft.org/articles/win/synctoy/images/echo.jpg (sorry, i can't publish urls from my noob account)
Basically, the DEST folder is never analysed and only the changes made on the SOURCE folder between date 1 and date 2 are repeated on DEST folder.
Assuming that i never make any manual modification to my DEST folder this is the perfect solution.
Actually, i presume iTunes acts exactly this way and that's why syncs are so faster with iphone and ipod.
Why do I want to replicate this mode?
My sdcard is a class2 and it's really sloooow. Each time I run rsync, it tries to analyse my mac partition (ssd) AND my sdcard, and compare the two directories....which takes at least 1h30 for my whole library even if only 2 files have been added to SOURCE folder between date 1 and date 2!!
SyncToy is a Windows program, I use a macbook
All my music library is on my mac partition
Note: for the moment i run synctoy through a virtual machine with my itunes library folder being shared. It's working but it's not the best solution...
So if there is any rsync guru here: do you think there is any possibility to recreate this 'echo mode' using rsync?
Is rsync really what you want, given you don't need any of the clever things it can do?
My shell scripting knowledge is very sparse, but wouldn't something like this do:
find $SourceDir -mtime +1 -mtime -4 -type f -exec cp {} $DestDir \;
where in this example it (maybe ) copies any file last modified between 2 & 4 days ago (inclusive & exclusive respectively) from $SourceDir to $DestDir.
NB This is untested, I'm clueless & it also might need to be a bit smarter to handle anything other than a flat file structure (in which case replace cp with something that will make each ancestor directory if absent from the destination & will then cp into that), so you have been warned.
[Edit:] Yes, you'd need something like cpio instead of cp. If you're interested search for info on "cpio -pd"
Otherwise I was thinking about making a script:
1. create a simple text file with lines like that "/full/path/to/file/filename.ext;SizeInByte;LastModificationTime" for each file file in SOURCE dir at date 1.
2. At date 2, create a new similar text file
3. Compare the 2 text files and keep only different lines
4. Differences coming from date1 text file are deleted from DEST dir and differences coming from date 2 file are copied/overwritten.
Does your command deletes files that have been deleted in SOURCE dir between the two dates?
Isn't cpio for archiving, like tar,gz2,etc...?
venezia64 said:
Otherwise I was thinking about making a script:
1. create a simple text file with lines like that "/full/path/to/file/filename.ext;SizeInByte;LastModificationTime" for each file file in SOURCE dir at date 1.
2. At date 2, create a new similar text file
3. Compare the 2 text files and keep only different lines
4. Differences coming from date1 text file are deleted from DEST dir and differences coming from date 2 file are copied/overwritten.
Does your command deletes files that have been deleted in SOURCE dir between the two dates?
Isn't cpio for archiving, like tar,gz2,etc...?
Click to expand...
Click to collapse
Deletes? No, that would be impossible if you aren't allowed to scan the destination folder, /unless/ you follow a plan such as you outline above. I didn't realise you required this.
cpio with the -p option (passthrough) is effectively just a copy command which maintains the directory hierarchy.
Your above strategy sounds best. For the stored records, I think I'd use:
find $Dir -type f -printf %p\\0%s\\0%A+\\0\\n > file.log
as you can rely on \0 not turning up in any of the fields it separates; the \n just added to tidy the log up should you wish to check it in a text editor. Oh & the %A+ date format so that if you need to order two dates a lexical comparison is adequate, whilst still being human readable.
That said, you don't appear to want to use the size or time fields, & not storing them would slightly simplify the code. <-- Sorry, that's rubbish on my part; how else would you notice when a file had changed.
Then probably compare them using diff:
diff -a file1.log file2.log > diff.log
Parsing diff.log should then be a trivial matter of checking each line in turn & for those commencing <, remove the named file; for those commencing >, copy the file over.
It's possible for a file to change resulting in both lines above appearing in diff.log for the one file. In that case you obviously need to process the removal first & it's not obvious to me if < will always appear in the log before >, so maybe safer to handle all the < removals first then handle the > copies.
OTOH if the above assumption is safe /&/ you modify the above to process the directories as well as files, you might be able to get away with making the diff.log parsing step a simple matter of applying a suitable regexp match & replace to each line in the diff.log, with the resulting output being a script that does the removals & copies. That'd be quite cool.
>> [That said, you don't appear to want to use the size or time fields, & not storing them would slightly simplify the code.]
Well, if I just add an albumart to my music file or change a tag, the filename will not necessarily change but the ModificationTime will, for sure.
I'll have a look at your shell commands.
Btw, thanks a lot for helping me
venezia64 said:
>> [That said, you don't appear to want to use the size or time fields, & not storing them would slightly simplify the code.]
Well, if I just add an albumart to my music file or change a tag, the filename will not necessarily change but the ModificationTime will, for sure.
Click to expand...
Click to collapse
I know. Stupid oversight on my part which I realised just before you posted
I edited my post to reflect this & had to resort to colouring the incorrect comment as xda-dev doesn't seem to support any strikethrough markup.
I managed do the script but there is still one issue: at the beginning, when I declare the path variable, I cannot include a path with one or more spaces. For the moment, I just created symbolic links.
Code:
left=$HOME"/Desktop/SyncAndroid/left"
right=$HOME"/Desktop/SyncAndroid/right"
temp=$HOME"/Desktop/SyncAndroid/temp"
rightSED=$(echo $right | sed -e 's/\//@@@/g')
leftSED=$(echo $left | sed -e 's/\//@@@/g')
mkdir "$temp"
cd $left
find . -type d -print | sort > $temp/LOG_leftD
cd $right
find . -type d -print | sort > $temp/LOG_rightD
cd $temp
diff -a LOG_leftD LOG_rightD > diffD
cat diffD | grep '^>' | sed -e 's/^\> \./rm -r \"'$rightSED'/' | sed -e 's/\(.\)$/\1\"/' | sed -e 's/@@@/\//g' | sed -e 's/\([$]\)/\\\1/g' > rmdir
cat diffD | grep '^<' | sed -e 's/^\< \./mkdir \"'$rightSED'/' | sed -e 's/\(.\)$/\1\"/' | sed -e 's/@@@/\//g' | sed -e 's/\([$]\)/\\\1/g' > mkdir
chmod +x rmdir
chmod +x mkdir
./rmdir
./mkdir
cd $left
find . -type f -ls | sed -e '/DS_Store/d' | sed -e 's/^ *[0-9]* *[0-9]* [-rwx]* *[0-9] [a-zA-Z]* *[a-z]* *//' | sort > $temp/LOG_left
cd $right
find . -type f -ls | sed -e '/DS_Store/d' | sed -e 's/^ *[0-9]* *[0-9]* [-rwx]* *[0-9] [a-zA-Z]* *[a-z]* *//' | sort > $temp/LOG_right
cd $temp
diff -a LOG_left LOG_right > diff
cat diff | grep '^>' | sed 's/^.*[0-9] \./rm -r \"'$rightSED'/' | sed -e 's/\(.\)$/\1\"/' | sed -e 's/@@@/\//g' | sed -e 's/\([$]\)/\\\1/g' > remove
cat diff | grep '^<' | sed 's/^.*[0-9] \.//' > copy0
cat copy0 | sed -e 's/\(.\)$/\1\"/' > copy1
cat copy0 | sed -e 's/[^/]\{1,\}\.*[a-zA-Z0-9]*$//' | sed -e 's/^/\"'$rightSED'/' | sed -e 's/@@@/\//g' | sed -e 's/\(.\)$/\1\"/' | sed -e 's/\([$]\)/\\\1/g' > copyPath
cat copy1 | sed -e 's/^/cp -p \"'$leftSED'/' | sed -e 's/@@@/\//g' | sed -e 's/\([$]\)/\\\1/g' > copy
paste copy copyPath > copy2
chmod +x remove
chmod +x copy2
./remove
./copy2
rm -r "$temp"
UPDATE 1: the code to take $ symbol (like in Ke$ha) in file path or filename
UPDATE 2: log files are sorted to fix some library updating bugs
NOTE: this scripts scans the right AND the left folder. In fact, I realised that scaning the music folder in the class2 sdcard only took 1-2seconds... Btw, I still don't understand why rsync takes so much time to sync the two folders (even if i don't ask for checksums comparisons)...
venezia64 said:
I managed do the script but there is still one issue: at the beginning, when I declare the path variable, I cannot include a path with one or more spaces. For the moment, I just created symbolic links.
<snip>
UPDATE 1: the code to take $ symbol (like in Ke$ha) in file path or filename
UPDATE 2: log files are sorted to fix some library updating bugs
NOTE: this scripts scans the right AND the left folder. In fact, I realised that scaning the music folder in the class2 sdcard only took 1-2seconds... Btw, I still don't understand why rsync takes so much time to sync the two folders (even if i don't ask for checksums comparisons)...
Click to expand...
Click to collapse
Glad you were able to devise a solution. Sed always gives me a headache as I can never remember what needs to be escaped & what doesn't. Clearly your grasp of shell scripting is better than mine.
Re the space in path issue, I take it it isn't adequate just to escape each space? The speed of rsync is puzzling. Is it definitely not connected to the sdcard? If you apply rsync to the same data, but with both directories on your pc, do you get the same performance differential?
I've tried doing this sync using rsync:
between 2 folders on my HDD, only modifications are done to the DEST folder
between a folder on my HDD and another one on my sdcard, each file file is copied all over again each time... which, obviously, takes time
I DID IT! If we look at rsync manual:
Code:
When comparing two timestamps, rsync treats the timestamps as
being equal if they differ by no more than the modify-window
value. This is normally 0 (for an exact match), but you may
find it useful to set this to a larger value in some situations.
In particular, when transferring to or from an MS Windows FAT
filesystem (which represents times with a 2-second resolution),
--modify-window=1 is useful (allowing times to differ by up to 1
second).
My sdcard partition is in FAT...
So the final rsync command which works really well
Code:
rsync -arv --delete --force --modify-window=1 "/Users/XXXXX/Music/iTunes/iTunes Music/Music/" "/Volumes/NO NAME/Music"
a for archive mode
r for recursive deletions (combined with --force)
v for more verbose (optional)
--delete to delete any file in DEST dir that is not in SRC dir
--force to force the deletion of not empty folders
--modify-window=1 to tolerate 1sec of difference between two corresponding files.
So forget about the previous script. This single commandline replaces it.
phewww
Awesome. Congratulations on figuring it out
With Android 4.2's multi-user stuff and the dreaded "0" folder, Google broke mounting of CIFS shares for good. (They basically implemented something utterly useless for my needs, and broke something I need on a daily basis.) Cyanogenmod fixed that for mounts outside the /storage hierarchy, but many apps can't go there. And with 4.2.2, Google made using adb on the device very annoying as well, which adds to problems for a workaround for mounting shares in a useful place.
So I looked around for various fixes for that issue and compiled those into one method that works perfectly well for me on a P6810 running the current PAC ROM 20130629. It should probably work for any other CM10.1 or PA3.6x-based ROM as well. A nls_utf8.ko module compiled for the Nexus 10 is included as well, which can be loaded via punchmod (at least in PAC), so you can access shares with Ümläüts in them.
Prerequisites: SManager, SMWidgets, and a working Busybox install (I use Stericson's Busybox Installer).
1.) Extract nls_utf8.ko and punchmod from the attached .zip file and put them in /system/lib/modules. You can put them in some other place as well, but my scripts need to be adjusted then. Next, give the files sufficient permissions:
Code:
chmod 644 /system/lib/modules/nls_utf8.ko
chmod 755 /system/lib/modules/punchmod
2.) Optional: if you want the module to auto-start, create a file called 00punch-nls_utf8 (or whatever you like) with the content below, put it in /etc/init.d and chmod it to 755. To check if nls-utf8 is loaded after a reboot, type lsmod in a terminal window or adb shell - it should display the module name. If punchmod auto-detection fails, read here how to get the right vermagic string for your device: http://forum.xda-developers.com/showthread.php?p=41920265#post41920265
Code:
#!/system/bin/sh
sleep 10
/system/lib/modules/punchmod /system/lib/modules/nls_utf8.ko ""
3.) Here's the script that mounts a CIFS share. Edit the settings on top and either put your correct vermagic string as well, or delete the "$VERMAGIC" part in line 22 and try your luck with auto-detection. I put some sanity checks in the script, but it might not work for all directories in /sdcard (or, /data/media/0, to call it by its 'real' name/location). If you don't want to use the nls-utf8 module, just delete lines 18-23, and iocharset=utf8 in line 67.
Name the script mount-music.sh or something, put the script wherever you like, open SManager and navigate to the script. Run the script with SU permissions. If everything goes well, it will ask you to accept an adb RSA key, and then adb mounts the share. You can also add a desktop widget for quick access to the script with SMWidgets.
Code:
#!/system/bin/sh
# Your settings here
IP="192.168.1.123"
SHARE="Music"
USER="yourusername"
PASS="yourpassword"
MOUNTPOINT="/data/media/0/cifs"
VERMAGIC="3.0.31-CM-ga034655-dirty SMP preempt mod_unload ARMv7 p2v8 "
# Load cifs (if it isn't already loaded)
if [ `lsmod | grep -o cifs` ] ; then
:
else
insmod /system/lib/modules/cifs.ko
fi
# Load nls_utf8 brute-force style (if it isn't already loaded)
if [ `lsmod | grep -o nls_utf8` ] ; then
:
else
/system/lib/modules/punchmod /system/lib/modules/nls_utf8.ko "" "$VERMAGIC"
fi
# Check if cifs $MOUNTPOINT dir exists, create it if not, give sufficient permissions
if [ -d $MOUNTPOINT ] ; then
:
else
mkdir $MOUNTPOINT
fi
chmod 755 $MOUNTPOINT
# Check if $SHARE directory is empty - if not, unmount the share
if [ "$(su root -c busybox ls -A $MOUNTPOINT/$SHARE 2> /dev/null)" == "" ] ; then
:
else
/system/xbin/busybox umount -l $MOUNTPOINT/$SHARE
fi
# Create $SHARE directory if necessary, give sufficient permissions
if [ -d $MOUNTPOINT/$SHARE ] ; then
:
else
mkdir $MOUNTPOINT/$SHARE
fi
chmod 755 $MOUNTPOINT/$SHARE
# Starting adb
adb kill-server
export HOME=/sdcard
# PORT=`getprop service.adb.tcp.port`
setprop service.adb.tcp.port 5555
adb start-server
cat /sdcard/.android/adbkey.pub >> /data/misc/adb/adb_keys
stop adbd
start adbd
adb connect localhost
# Make sure we only use the first device (sometimes there is more than one)
SERIAL=`adb devices | head -n2 | tail -n1 | cut -f1`
if [ "$SERIAL" = "" ] ; then
echo "ERROR: Could not find ADB device.";
fi
# Mounting share via adb
echo Mounting share via adb...
adb -s $SERIAL shell su root -c /system/xbin/busybox mount -t cifs //$IP/$SHARE $MOUNTPOINT/$SHARE -o user=$USER,pass=$PASS,iocharset=utf8,cache=none,directio,CIFSMaxBufSize=130048,rw,file_mode=0777,dir_mode=0777,uid=1015,gid=1015
# If you started adb, then stop it here for security
adb disconnect localhost
stop adbd
PORT=`getprop service.adb.tcp.port`
setprop service.adb.tcp.port $PORT
start adbd
# Show some results
RESULT=`mount | grep $MOUNTPOINT/$SHARE`
if [ "$RESULT" = "" ] ; then
echo "FAILED! //$IP/$SHARE could not be mounted."
else
echo "SUCCESS! //$IP/$SHARE has been mounted on $MOUNTPOINT/$SHARE."
fi
echo
echo All done. You may close this script window now.
4.) This is the script to unmount the share. Name it unmount-music.sh or similar, and edit your settings at the beginning of the script. Again, run it in SManager with SU permissions, and add a desktop widget with SMWidgets if you like.
Code:
#!/system/bin/sh
# Your settings here
SHARE="Music"
MOUNTPOINT="/data/media/0/cifs"
# Check if $SHARE directory exists
if [ -d $MOUNTPOINT/$SHARE ] ; then
# Check if $SHARE directory is empty - if not, unmount the share
if [ "$(su root -c busybox ls -A $MOUNTPOINT/$SHARE 2> /dev/null)" == "" ] ; then
echo "$MOUNTPOINT/$SHARE is empty."
else
su root -c busybox umount -l $MOUNTPOINT/$SHARE
fi
fi
# Show some results
RESULT=`mount | grep $MOUNTPOINT/$SHARE`
if [ "$RESULT" = "" ] ; then
echo "SUCCESS! $MOUNTPOINT/$SHARE has been unmounted."
else
echo "FAILED! $MOUNTPOINT/$SHARE could not be unmounted."
fi
echo
echo All done. You may close this script window now.
Hope this helps some of you. It sure stumped me how incredibly convoluted this simple and much needed procedure has become on Android 4.2.2... just because of that multi-account stuff and new restrictions on adb.
Credits:
Script bits and nls_utf8.ko by H3g3m0n
More script bits by dafunk60 and jmtw000
Punchmod by Jann Horn
Punchmod info by idcrisis
Thanks man, looks awesome, I am going to try it on my 6800.
Do you know how to mount ext4 or ntfs sdcard on CM10.1 based roms?
Removed the original post, still trying to make it work!
I checked vermagic in some existing ko files and found that it is exactly the same as yours.
Sent from my GT-p6800 using Tapatalk HD
m0bster said:
Removed the original post, still trying to make it work!
I checked vermagic in some existing ko files and found that it is exactly the same as yours.
Sent from my GT-p6800 using Tapatalk HD
Click to expand...
Click to collapse
You can try loading the UTF8 module without vermagic string as well, it should work on PAC. Or, make sure you have a blank space at the end of the vermagic string, it is required. Maybe read the short tutorial here: http://forum.xda-developers.com/showthread.php?p=41920265#post41920265
dfkt_ said:
You can try loading the UTF8 module without vermagic string as well, it should work on PAC. Or, make sure you have a blank space at the end of the vermagic string, it is required. Maybe read the short tutorial here: http://forum.xda-developers.com/showthread.php?p=41920265#post41920265
Click to expand...
Click to collapse
I finally managed to mount cifs shares, after loading original cifs and punmode load of nls_utf8.ko.
But I mounted it with cifsmanager. The script always shows "failure to mount" !
Hey guys, I've just decided to get inside of the Android development world. Anyways, I'm mostly a fan of Linux, so I've made my first Linux script, I know it's simple, but I've never interacted with the user before, or picked an output. When I knew that Android made use of Linux in its core, my interest on working inside it grew up faster than ever.
In my script, firstly, you type the filename. For compatibility reasons, it'll everytime save to /sdcard/filename. Then, it will ask you how many times you want the script to log everything, and lately, every how many seconds should it save everything to the file.
The script will save, before every entry at the log, the screen status (if it's on or off), that can help to, for example, if your device has been lagging as a result of a bad screen driver or UI error.
Last thnig: it also works on recovery (maybe it give some error, and if no dumpsys is present on your recovery's ramdisk, it'll save everything as OFF).
Remember to send it anywhere excepting from /storage and any sbudir, it won't work as the Android permissions system blocks script execution on those dirs.
Code:
clear
# Wipe $logfile content in /sdcard (if any existing)
echo "Please type the filename where you want to save the log: "
read -r logfile
echo "" > /sdcard/$logfile
echo "Deleted previous log."
echo "SCREEN | CPU" >> /sdcard/$logfile
#cont=60
cont=1
#SHOWLOG=X
echo "Please type how many times the cycle must repeat: "
read -r maxcont
echo "Please type how many seconds will the interval take: "
read -r maxmin
clear
# Here it counts how many times the logging script has been ran
while [ $cont -lt $((maxcont + 1)) ];
do
# That nested while takes control of how many seconds have passed in the current cycle
while [ "$contmin" -lt "$maxmin" ];
do
contmin=$((contmin + 1))
sleep 1
clear
echo "C: $cont/$maxcont - T: $contmin/$maxmin"
done
# echo "$maxmin seconds passed."
# Logs the fist line of "busybox top" to a var
LOGVAR=$(busybox top -d 1 -n 1 | grep "idle")
# Logs the screen state (on or off) to another var
SCREENSTATE=$(dumpsys input_method | grep mScreenOn | tr -dc '[:alnum:]\n\r' | tr '[:upper:]' '[:lower:]')
# Saves a string that tells what it should expect if the screen is on
SCREENSTATEON="msystemreadytruemscreenontrue"
if [ "$SCREENSTATE" == "$SCREENSTATEON" ]
then
echo "ON: $LOGVAR" >> /sdcard/$logfile
else
echo "OFF: $LOGVAR" >> /sdcard/$logfile
fi
cont=$((cont + 1))
contmin=0
done
echo "Show logged values? [Y/N]"
read -r SHOWLOG
case $SHOWLOG in
[yY] | [yY][Ee][Ss] )
cat /sdcard/$logfile
;;
[nN] | [n|N][O|o] )
echo "Ok, goodbye. You'll see a what you've asked me to log in /sdcard/$logfile";
exit 1
;;
*) echo "You must write Y or N to continue."
;;
esac
exit 0
For example, save it as script.sh (remember to save it using Unix codifcation, if you're running Windows, else, it'll tell you that it's unable to find many binaries, as long as Linux isn't ok with CR+LF and requires LF as newline). Then, send it to the sdcard (over USB or adb push script.sh /sdcard/) , now, run ADB shell. Type the following commands:
Code:
su -c "cp -f /sdcard/script.sh /data/local/;chmod 777 /data/local/script.sh"
Then, if you want to run it:
Code:
su
sh /data/local/script.sh
That's all, I hope you enjoy it and not so many people kills me for explaining and feeling so grateful for that simple script.
Native ARM/static Linux binaries
(for all ARMv7+ compatible platforms)
Open-source Linux binaries that are either not available on Android (e.g. in Termux)
or make sense to be statically compiled (e.g. to run in TWRP/recovery for data recovery).
These are root tools and might damage your device severely. Use at your own risk. I take no responsibility whatsoever. If in doubt don't use them.
Minimum CPU: ARMv7/vfpv3-d16. Compiled against musl-libc/Android Kernel 3.4. Binaries are static, bionic/libc independent and should run on Android, TWRP, emulator or any other compatible ARM device. Musl is patched (info)(info2)(patch file: patch -p0 -u -b -i musl-android-smp.patch) to iterate CPU cores by /proc/stat instead of _SC_NPROCESSORS_CONF/sched_getaffinity to prevent false detection due to ARM cpu core powersaving (permanently turning cores on/off). This should report CPU cores more reliably to multithreading apps.
Example instructions how to build EncFS can be found here.
Some Cryptsetup compile recipes are here.
Changelog:
20190923 - f2fs-tools added
20190915 - dislocker, ntfs-3g, mount.exfat-fuse added
20190910 - VeraCrypt v1.24-b5 added
20191215 - musl smp patch added
20191224 - hstr v2.2.0 updated
20191225 - Testdisk, PhotoRec v7.2-wip-dec2019 updated
20200103 - tar v1.32 updated (with selinux, acl, xattr support)
20200513 - Cryptsetup v2.3.2 added
20200518 - fscrypt 0.2.7, strace56(aarch64) added
20200525 - p7zip v17.01 added
20200603 - parted v3.3 added
20200606 - fxz v1.1.0alpha added
20201212 - ddrescue v1.25 added
20201212 - Cryptsetup v2.3.4 updated
20210113 - f2fs-tools updated to v1.14.0
20210125 - Several tools compiled by @Borovets. See 'Misc' tools.
20210413 - Cryptsetup v2.3.5 updated
20210916 - Cryptsetup v2.4.1 updated. Thx to @misterhsp.
20211108 - rsync v3.2.3 updated
20211118 - Cryptsetup v2.4.2 updated. Thx to @misterhsp.
20220103 - mmc-utils added
20220106 - More tools from @Borovets. See spoiler.
Spoiler
bash-5.1.16-[1]-[2022.01.05].tar.gz
openssl3-3.0.1-[2021.12.14]-static.tar.gz
tree-2.0.0-[2021.12.23]-static.tar.gz
e2fsprogs-1.46.5-[2021.12.31]-static.tar.gz
openssl-1.1.1-m-[2021.12.15]-static.tar.gz
libsqlite-3.37.1-[2021.12.30]-static.tar.gz
ldns-host-1.7.1-[2021.12.30]-static.tar.gz
bootimg-info-2.0-[2021.12.18]-static.tar.gz
bc-5.2.1-[2021.12.29]-static.tar.gz
openssl3-tool-3.0.1-[2021.12.14]-static.tar.gz
openssl-tool-1.1.1-m-[2021.12.15]-static.tar.gz
sqlite-3.37.1-[2021.12.30]-static.tar.gz
stunnel-5.61-[2021.12.17]-static.tar.gz
toybox-0.8.6-borovets-295-applets-[2021.12.30]-static.tar.gz
unrar-6.10-beta-3-[2021.12.11]-static.tar.gz
zstd-1.5.1-[2021.12.22]-static.tar.gz
20220107 - parted v3.4 updated
20220113 - cryptsetup v2.4.3 updated. Thx to @misterhsp.
20220114 - gptfdisk v1.0.8 added
20220212 - tar v1.34 updated
20220622 - gptfdisk v1.0.9 (armv7) added
20220724 - dialog v1.3 added
20220728 - f2fs tools v1.15.0 updated
20220730 - cryptsetup v2.5.0 updated. Thx to @misterhsp.
20220806 - 7z-zstd v22.01 added. Thx to @xenosaur
20220910 - rsync v3.2.6 updated
20220913 - htop v3.2.1 added
20220913 - gocryptfs v2.3 updated. Thx to @misterhsp
20220922 - veracrypt v1.25.9 updated
20220924 - fdisk v2.38.1 and file v5.43 added
20221129 - cryptsetup v2.6.0 updated. Thx to @misterhsp
20221213 - f2fs tools v1.15.0 fixed (uuid.h missing)
20230215 - cryptsetup v2.6.1 updated. Thx to @misterhsp
20230307 - gocryptfs v2.3.1. Thx to @misterhsp
Data recovery tools:
- PhotoRec 7.2 - PhotoRec is file data recovery software designed to recover lost files including video, documents and archives from hard disks, CD-ROMs, and lost pictures (thus the Photo Recovery name) from digital camera memory. PhotoRec ignores the file system and goes after the underlying data, so it will still work even if your media's file system has been severely damaged or reformatted.
- Testdisk 7.2 - Recover lost partitions and partition tables. For external sdcards. Never use it on internal mmc unless you know what you're doing.
- ext4magic 0.3.2 (with supplementary gnu date binary that can handle relative time like 'date -d "-20minutes" +%s')
- fidentity - A little utility sharing PhotoRec signature database. It identifies the type of data contained in a file and reports the extension as seen by PhotoRec.
- debugfs - Might be helpful on ext2 systems or other stuff.
- strace 4.20 - For debugging. Mainly to catch syslog messages (as Android has no traditional /dev/log buffer).
- strace 5.6 - For aarch64.
- ddrescue v1.25 - Data recovery tool for block devices with errors.
Compression tools:
p7zip v17.01 (fork) - (Download) A new p7zip fork with additional codecs and improvements
pixz - Parallel, indexed xz compressor
xz - Multicore aware version of xz/lzma (use --thread=0)
tar v1.32 - Tar provides the ability to create tar archives, as well as various other kinds of manipulation. Download below. More builds from @mirfatif here.
fxz - (Download) FXZ Utils is a fork of XZ Utils. It adds a multi-threaded radix match finder and optimized encoder.
Misc:
- hexcurse v1.60.0 - Hexcurse is a curses-base hex editing utility that can open, edit, and save files, editing both the hexadecimal and decimal values. 'ncurses' ui layout depends on TERM env variable. Change temporary with eg. 'TERM=xterm-256color hexcurse <file>'. See /system/etc/terminfo for possible terminals (xterm-256color, linux..).
- nethogs v0.8.5 - ncurse/nettop-like per-app separated speedmeter and traffic counter supporting high refresh rate. Try 'nethogs -d0' (speedmeter) or 'nethogs -v1' (traffic counter).
- rsync v3.2.3 - rsync is an open source utility that provides fast incremental file transfer. (--with-rsyncd-conf=/data/etc/rsyncd.conf)
- smbnetfs v0.6.1 - SMBNetFS is a Linux/FreeBSD filesystem that allow you to use samba/microsoft network in the same manner as the network neighborhood in Microsoft Windows. More info see below.
- progress v0.14 - Linux tool to show progress for cp, mv, dd, ... (formerly known as cv). Download here.
- archivemount (20180801) - A fuse filesystem for mounting archives in formats supported by libarchive. Download here.
- squashfuse v0.1.103 - FUSE filesystem to mount squashfs archives Download here.
- FuseISO - FuseISO is a FUSE module to mount ISO filesystem images (.iso, .nrg, .bin, .mdf and .img files). It currently support plain ISO9660 Level 1 and 2, Rock Ridge, Joliet, and zisofs. Download here.
- HSTR v2.2.0 - HSTR (HiSToRy) is a command line utility that brings improved Bash/zsh command completion from the history. It aims to make completion easier and more efficient than Ctrl-r. (If history is empty try setting HISTFILE in /system/etc/bash/bashrc e.g. export HISTFILE=/data/.bash_history).
- GNU screen, tmux - Thanks to @mirfatif.
- dislocker, ntfs-3g, mount.exfat-fuse - Thanks to @mirfatif.
- f2fs-tools - Thanks to @mirfatif. Update: v1.14.0 here.
- parted v3.3 - GNU Parted (the name being the conjunction of the two words PARTition and EDitor) is a free partition editor, used for creating and deleting partitions. Note: It might be useful to partition external sdcards (e.g. to limit adoptable storage). I do not recommend to use it on internal memory. It might brick your phone.
- Several tools compiled by @Borovets
Spoiler: Borovets tools
Borovets tools 2021.01.25
arptables-0.0.5-[2021.01.17]-static.zip
autoflushtest-1.0-[2021.01.14]-static.zip
btrfs-compsize-1.3-[build-2]-[2020.12.27].zip
btyacc-3.0-[2021.01.18]-static.zip
c-blosc-1.21.1-development-[2020.12.22].zip
c-blosc2-2.0.0-beta-6-development-[2020.04.21].zip
cabextract-1.9.1-[2021.01.08]-static.zip
compsize-1.3-[2021.01.07]-static.zip
convert-color-space-0.1-[2021.01.18]-static.zip
cpustat-0.02.13-[2021.01.13]-static.zip
doxygen-1.9.2-[2021.01.17]-static.zip
ed-1.17-[2021.01.11]-static.zip
hello-2.10-[2021.01.08]-static.zip
htop-3.0.5-[2021.01.13]-static.zip
ipcalc-ng-1.0.0-[2020.12.28]-static.zip
iw-5.9-[2021.01.08]-static.zip
libsqlite-3.34.1-[2021.01.20].zip
libtar-1.2.20-[2021.01.16]-static.zip
m5-1.0-[2020.12.31]-static.zip
sqlite-3.34.1-[2021.01.20]-static.zip
Borovets tools 2021.01.27
lcab-1.0-beta-12-[2021.01.17].zip
memdump-1.01-[2021.01.25].zip
memdumper-0.4-[2021.01.25].zip
memtester-4.5.0-[2021.01.09].zip
tcpdump-4.99.0-[libcap-1.9.1]-[2021.01.05].zip
wget2-1.99.2-[2020.12.12].zip
wolfssl-4.5.0-[2020.12.12].zip
xfsprogs-5.10.0-[2021.01.01].zip
Crypttools:
(These crypttools are mostly frontend tools for the main backend that resides in the kernel. If your kernel hasn't been configured accordingly at compile time you might not be able to use all features.)
Cryptsetup v2.3.5 - (Download) Cryptsetup is an utility used to conveniently setup disk encryption based on DMCrypt kernel module. These include plain dm-crypt volumes, LUKS volumes, loop-AES and TrueCrypt (including VeraCrypt extension) format.
eCryptfs-utils v111 - Frontend tools for the enterprise cryptographic filesystem for Linux. That's what Android/Google use for encryption. It's file-based (no container) and mounting can be automated by Termux widget. Needs shared libraries but is still portable. See notes below.
EncFS v1.9.5 - EncFS provides an encrypted filesystem in user-space. It runs in userspace, using the FUSE library for the filesystem interface.
gocryptfs - An encrypted overlay filesystem written in Go. Download here. Thanks to @mirfatif.
VeraCrypt - VeraCrypt is a free open source disk encryption software. Download here. Thanks to @mirfatif.
fscrypt 0.2.7 - (Download) fscrypt is a high-level tool for the management of Linux filesystem encryption. Needs at least kernel 4.1.
Crypttools info:
Cryptsetup:
General Notes:
- Features like TrueCrypt, VeraCrypt and LUKS2 need 'userspace crypto api' enabled in kernel. Most Android kernels are probably not configured for that and you have to recompile your kernel or contact your kernel maintainer. For kernel 3.4 you need this:
Code:
CONFIG_CRYPTO_USER=y
CONFIG_CRYPTO_USER_API=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRYPTO_USER_API_SKCIPHER=y
- If 'cryptsetup benchmark' is incomplete and says 'userspace crypto api not available' you might be affected. You can still use LUKS1 though. A full benchmark looks like this:
Code:
# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1 249186 iterations per second for 256-bit key
PBKDF2-sha256 327680 iterations per second for 256-bit key
PBKDF2-sha512 58829 iterations per second for 256-bit key
PBKDF2-ripemd160 227555 iterations per second for 256-bit key
PBKDF2-whirlpool 33539 iterations per second for 256-bit key
argon2i 4 iterations, 208288 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id 4 iterations, 207817 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
# Algorithm | Key | Encryption | Decryption
aes-cbc 128b 77.8 MiB/s 88.4 MiB/s
serpent-cbc 128b N/A N/A
twofish-cbc 128b 58.5 MiB/s 61.9 MiB/s
aes-cbc 256b 61.5 MiB/s 68.4 MiB/s
serpent-cbc 256b N/A N/A
twofish-cbc 256b 58.5 MiB/s 61.8 MiB/s
aes-xts 256b 95.1 MiB/s 86.9 MiB/s
serpent-xts 256b N/A N/A
twofish-xts 256b 60.0 MiB/s 61.8 MiB/s
aes-xts 512b 74.1 MiB/s 67.2 MiB/s
serpent-xts 512b N/A N/A
twofish-xts 512b 60.3 MiB/s 62.0 MiB/s
LUKS:
Code:
** 10MB test image (luks.img) **
dd if=/dev/zero of=luks.img bs=1M count 10M
cryptsetup luksFormat luks.img
cryptsetup open luks.img myluks
mke2fs -t ext4 /dev/mapper/myluks
mkdir luks
mount /dev/mapper/myluks luks
** luks folder is ready here **
umount luks
cryptsetup close myluks
- If standard luksFormat cipher (aes-xts-plain64) doesn't work (not supported by your kernel) you can try one of the more compatible ciphers:
Code:
cryptsetup luksFormat -c cbc-essiv:sha256 luks.img myluks
cryptsetup luksFormat -c aes-plain luks.img myluks
- For LUKS2 (experimental) use:
Code:
cryptsetup luksFormat --type luks2 luks.img
- Use "cryptsetup -v --debug" for more verbose output (debugging). In case of errors.
Veracrypt:
Code:
cryptsetup open --type tcrypt --veracrypt veracrypt.tc myvera
cryptsetup status myvera
mkdir /data/myvera
mount /dev/mapper/myvera /data/myvera
umount /data/myvera
cryptsetup close myvera
- Use container from desktop system (created with real Veracrypt)
- "veracrypt.tc" is the veracrypt container name
- "myvera" is an arbitrary name (handle)
- Use "cryptsetup -v --debug" for more verbose output (debugging). In case of errors.
eCryptfs-utils:
General Notes:
These tools are not built statically as they explicitly rely on 'dlopen' (plugin system). Instead they are compiled with relative rpaths (./libs). That means dependencies (libraries in subfolders) must be present in the binaries folder and you have to be in the binaries folder itself (with 'cd') before invoking any binary. By this the binaries are still portable (system independent) as long as the subfolders are present. I've put the files into a tar.gz archive so permissions should be set +x already. Extract the archive into /data/local/bin for 'Example' below.
Code:
mkdir -p /data/local/bin
cd /data/local/bin
tar xf crypttools.armv7.20180204.tar.gz
cd ecryptfs
./ecryptfs-stat --help
More info: ArchLinux Wiki
Example:
Tested on /sdcard based on FUSE filesystem. sdcardfs untested. Might need selinux permissive.
We create a folder /sdcard/pics that can be enabled (files present) or disabled (no files present) by a click on a widget button (Termux script) and entering our password. The encryption is done on a per-file basis. The actual files are stored encrypted in /sdcard/efs/pics.
- You might need SuperSU or Magisk Superuser for 'su -mm'. That makes sure that all apps can see the mounted folder (mount namespace separation).
- Busybox needed
- Install Termux and Termux:Widget from F-Droid or Playstore
- Start it and enter:
Code:
pkg upgrade
pkg install tsu
exit
- Create script /data/data/com.termux/files/home/.shortcuts/efs-pics.sh and make sure permissions(700) and owner (take from parent folder) are correct.
Code:
#!/system/xbin/bash
su -mm -c "/system/xbin/bash -c /data/local/scripts/$(basename "$0")"
- Create script /data/local/scripts/efs-pics.sh (770/root):
Code:
#!/system/xbin/bash
set -e
PATH=$PATH:/data/data/com.termux/files/usr/bin
# Necessary because rpaths are relative
cd /data/local/bin/ecryptfs
# /data/myefskey contains the salted key.
# Don't forget to make a backup.
# Without it encrypted data is lost.
function enter_passphrase {
read -p "Enter passphrase: " passphrase
sig=$(printf "%s" "$passphrase" | ./ecryptfs-insert-wrapped-passphrase-into-keyring /data/myefskey -) || exit
sig=$(echo $sig | cut -d "[" -f2 | cut -d "]" -f1)
}
CPATH1="/data/media/0/efs/pics"
CPATH2="/data/media/0/pics"
if ! mountpoint -q ${CPATH2}; then
enter_passphrase
echo ""
mount -t ecryptfs -o ecryptfs_sig=$sig,ecryptfs_fnek_sig=$sig,ecryptfs_cipher=aes,ecryptfs_key_bytes=16 ${CPATH1} ${CPATH2} || (echo "$(basename "$0") mount failed!"; exit)
./keyctl clear @u
echo "$(basename "$0") mount successful! :)"
else
umount ${CPATH2} || (echo "$(basename "$0") umount error $? :("; exit)
echo "$(basename "$0") umount successful :)"
fi
# uncomment to force-close Termux window
# killall com.termux
- If your rom uses encryption already (/data/data) beware the './keyctl clear @u' command. It might flush *all* keys in the kernel including the Android encryption one (i'm not sure). This might lead to unpredicted behavior. Either comment it out (then your once injected key remains in the kernel keystore and someone could simply remount your folder without passphrase) or make yourself familiar with the keyctl command and handle it yourself. My phone is not encrypted so i cannot help here.
- Create random keyfile (/data/myefskey) and wrap it with passphrase. This might need 1-2 minutes depending on your devices entropy pool (/dev/random). Backup this key (/data/myefskey). Without it your encrypted data is lost. And don't forget the trailing '-' (minus) at the end of the line, it's important.
Code:
cd /data/local/bin/ecryptfs
read -p "Enter passphrase: " passphrase; printf "%s\n%s" $(busybox od -x -N 100 --width=30 /dev/random | head -n 1 | busybox sed "s/^0000000//" | busybox sed "s/[[:space:]]*//g") "${passphrase}" | ./ecryptfs-wrap-passphrase /data/myefskey -
- Create folders:
Code:
mkdir -p /sdcard/efs/pics /sdcard/pics
- Create Widget (Termux) and select 'efs-pics.sh'.
- Start it and enter your passphrase (you used above). If everything goes well (it will tell you) you can place files into /sdcard/pics and scrambled files should come up in /sdcard/efs/pics. Never write into /sdcard/efs/pics directly.
- Activate widget again. /sdcard/pics should get emptied.
- Optional: You can set /data/media/0/efs/pics to 700/root so no one can access/see the encrypted data.
SMBNetFS info:
Note: The library paths are relative. You need to be in the folder (with 'cd') to spawn the executable (./smbnetfs). You can extract the archive wherever you want though as far as the file/folder structure remains intact.
Example:
Code:
mount -o remount,rw /
mkdir -p /data/local/bin /mnt/cifs
mount -o remount,ro /
tar xf smbnetfs.tar.gz -C /data/local/bin
cd /data/local/bin/smbnetfs
export HOME=/data/local/bin/smbnetfs/home
* enter your smb credentials into smbnetfs/home/.smb/smbnetfs.auth (eg. auth "192.168.1.2" "${user}" "${pass}")
./smbnetfs /mnt/cifs
cd /mnt/cifs/192.168.1.2/${share}
I think it usually should list the samba environment in /mnt/cifs but i'm not sure which prerequisites are necessary for that (edit: maybe it needs real workgroup/hostname instead of IPs). If nothing comes up this should work:
The folder 192.168.1.2/${share} is unreachable by Androids folder picker (unless you can enter the path manually). So either pre-create the folder structure beforehand (mkdir -p /mnt/cifs/192.168.1.2/${share}) and add/register the folder to your app by folder picker (eg. MXPlayer) and then overmount that with the actual ${share}. Or bindmount the folder:
Code:
mount --bind /mnt/cifs/192.168.1.2/${share} /mnt/cifs2
Edit: Another option is to let smbnetfs create a static link (actually a symbolic link) to the share in the mountpoint root (/mnt/cifs). Its not as robust as the bindmount though. MXPlayer doesn't find any files (even though the folder picker shows the folders properly).
Code:
echo "link myfiles 192.168.1.2/${share}" > /data/local/bin/smbnetfs/home/.smb/smbnetfs.host
chmod 700 /data/local/bin/smbnetfs/home/.smb/smbnetfs.host
I've noticed that MXPlayer shows the samba folders just for a glimpse of a second. But if you enter one of the local folders and then go back all samba folders are there. Not sure why this is happening or maybe its just my system.
Edit2: Not yet tested but.. check the permissions. Its possible that SMBNetFS mounts with 755 or something. That's inaccessible for Android apps. Try this:
Code:
./smbnetfs -o umask=000,noatime,noexec,nodev,nosuid /mnt/cifs
Samba 4.8.3 configuration:
Code:
_idmap_modules=idmap_rid,idmap_hash,idmap_tdb2
_pdb_modules=pdb_tdbsam,pdb_smbpasswd,pdb_wbc_sam,pdb_samba4
_auth_modules=auth_unix,auth_wbc,auth_server,auth_netlogond,auth_script,auth_samba4
waf configure --prefix=/tmp/myout \
-C \
--sysconfdir=./conf/etc/samba \
--with-configdir=./conf/etc/samba \
--localstatedir=./conf/var \
--libexecdir=./conf/usr/lib \
--enable-fhs \
--with-lockdir=./conf/var/cache/samba \
--with-piddir=./conf/run/samba \
--with-logfilebase=./conf/var/log/samba \
--without-pam \
--without-systemd \
--without-ads \
--with-shared-modules=$_idmap_modules,$_pdb_modules,$_auth_modules \
--disable-cups \
--without-gettext \
--bundled-libraries=NONE,com_err,ldb,uid_wrapper,resolv_wrapper,socket_wrapper,nss_wrapper,ntdb,roken,wind,hx509,asn1,heimbase,hcrypto,krb5,gssapi,heimntlm,hdb,kdc,cmocka,talloc,tdb,pytdb,ldb,pyldb,tevent,pytevent \
--disable-rpath-install \
--disable-python --without-ad-dc --without-acl-support --without-ldap \
--hostcc=/usr/bin/gcc \
--cross-compile --cross-execute='qemu-arm -L /media/devpart/qemu/root'
waf build -j4
waf install
Compression tools added.
Next are crypttools (ecryptfs-utils, cryptsetup).
DualJoe said:
Compression tools added.
Next are crypttools (ecryptfs-utils, cryptsetup).
Click to expand...
Click to collapse
Please add ecryptfs-simple
xyne.archlinux.ca/projects/ecryptfs-simple
Thanks.
Ecryptfs-simple is not POSIX compliant. It relies on an argv interface (to parse command-line parameters) that is a GNU extension that musl doesn't support.
The original eCryptFS is simple enough anyway (and its the upstream project). I will provide a quickstart example and a one button GUI controlled solution (Termux widget) to handle it.
Please to add gifsicle,
http://github.com/kohler/gifsicle
Thanks.
I only have gifsicle. The other ones are too complex for my setup atm.
DualJoe said:
I only have gifsicle. The other ones are too complex for my setup atm.
Click to expand...
Click to collapse
Thank you very much.
Please help me again to build giflossy (fork of gifsicle).
I really need it to compress (--lossy=N) the Gif file to be smaller.
https://github.com/kornelski/giflossy
Thanks.
Do you use them directly on your phone for web postings or something? What's your use case to not prefer a desktop system for this?
DualJoe said:
Do you use them directly on your phone for web postings or something? What's your use case to not prefer a desktop system for this?
Click to expand...
Click to collapse
I use it directly on the phone, for learning purposes.
Using it on the phone is so handy that it can be easily used anywhere.
Thanks.
Please help me again to build lbzip2
http://lbzip2.org/
Thanks.
Here it is.
DualJoe said:
Compression tools added.
Next are crypttools (ecryptfs-utils, cryptsetup).
Click to expand...
Click to collapse
When will Crypttools be released.
I've waited for the major update of cryptsetup. Its out now indeed. I should get it up this week.
Crypttools and quickstart tutorials added.
Mountpoint is not writable (eCryptfs)
DualJoe said:
Crypttools and quickstart tutorials added.
Click to expand...
Click to collapse
Can't write to mountpoint.
# touch /sdcard/pics/test
touch: /sdcard/pics/test: Permission denied
# cp file /sdcard/pics
cp: can't create '/sdcard/pics/file': Permission denied
buengeut said:
touch: /sdcard/pics/test: Permission denied
Click to expand...
Click to collapse
What are your permissions?
Code:
# stat /data/media/0/pics
Access: (775/drwxrwxr-x) Uid: (1023/media_rw) Gid: (1023/media_rw)
# stat /data/media/0/efs
Access: (775/drwxrwxr-x) Uid: (1023/media_rw) Gid: (1023/media_rw)
# stat /data/media/0/efs/pics
Access: (775/drwxrwxr-x) Uid: (1023/media_rw) Gid: (1023/media_rw)
How does your mount look like?
Code:
# mount |grep pics
/data/media/0/efs/pics on /data/media/0/pics type ecryptfs (rw,relatime,ecryptfs_fnek_sig=56b1f3c519fb3412,ecryptfs_sig=56b1f3c519fb3412,ecryptfs_cipher=aes,ecryptfs_key_bytes=16)
Is /sdcard linked?
Code:
# ls -l /sdcard
lrwxrwxrwx 1 root root 21 May 10 1973 /sdcard -> /storage/self/primary
What Android version and kernel do you have?
DualJoe said:
What are your permissions?
Code:
# stat /data/media/0/pics
Access: (775/drwxrwxr-x) Uid: (1023/media_rw) Gid: (1023/media_rw)
# stat /data/media/0/efs
Access: (775/drwxrwxr-x) Uid: (1023/media_rw) Gid: (1023/media_rw)
# stat /data/media/0/efs/pics
Access: (775/drwxrwxr-x) Uid: (1023/media_rw) Gid: (1023/media_rw)
How does your mount look like?
Code:
# mount |grep pics
/data/media/0/efs/pics on /data/media/0/pics type ecryptfs (rw,relatime,ecryptfs_fnek_sig=56b1f3c519fb3412,ecryptfs_sig=56b1f3c519fb3412,ecryptfs_cipher=aes,ecryptfs_key_bytes=16)
Is /sdcard linked?
Code:
# ls -l /sdcard
lrwxrwxrwx 1 root root 21 May 10 1973 /sdcard -> /storage/self/primary
What Android version and kernel do you have?
Click to expand...
Click to collapse
Android 6.0 kernel 3.18.14
/sdcard is symlink to /mnt/sdcard, i changed /sdcard to /mnt/sdcard
Code:
# mount -t ecryptfs
/mnt/sdcard/efs/pics on /mnt/sdcard/pics type ecryptfs (rw,relatime,ecryptfs_fnek_sig=1b77138d91206e66,ecryptfs_sig=1b77138d91206e66,ecryptfs_cipher=aes,ecryptfs_key_bytes=16)
Code:
# stat /mnt/sdcard/pics
Access: (775/drwxrwxr-x) Uid: (1000/ system) Gid: (1015/sdcard_rw)
# stat /mnt/sdcard/efs
Access: (775/drwxrwxr-x) Uid: (1000/ system) Gid: (1015/sdcard_rw)
# stat /mnt/sdcard/efs/pics
Access: (775/drwxrwxr-x) Uid: (1000/ system) Gid: (1015/sdcard_rw)
Code:
# touch /mnt/sdcard/pics/test
touch: /mnt/sdcard/pics/test: Permission denied
What about the permissions of /data/media/0 folders? That's the most important part.
If your sdcard is not at /data/media/0 you probably don't have a multiuser environment (older phone?) and /mnt/sdcard is probably a real partition. This is early Kitkat partition layout (/sdcard and /data have separate partitions). On later systems both are on /data partition and /sdcard is abstracted by a FUSE file system that would automatically set the proper permissions whenever you write something to it (even as root).
In case you are on an old layout you would need to set proper permissions to /sdcard/pics and /sdcard/efs yourself. Just take a look at the other folders with 'ls -l /mnt/sdcard' and set accordingly. You would also need to change /data/media/0 to /mnt/sdcard in the script.
What do you get with this?
Code:
# mount |grep sdcard
# mount |grep storage
What phone is it? Kernel 3.18 doesn't sound all too old.
Edit: Another theory is your internal sdcard is scardfs or something. If so, it might break "stacking" folders (mount over). Try to use /data/pics and /data/efs/pics as a test.
It works in Permissive mode (setenforce 0)
I need Busybox with SELinux-enabled and use it to set it to Permissive mode
Code:
# busybox getenforce
Enforcing
# busybox setenforce 0
# busybox getenforce
Permissive
And then execute the efs-pics.sh and test it
Code:
# cp file /mnt/sdcard/pics ; echo $?
[b]0[/b]
# ls /mnt/sdcard/pics
[b]file[/b]
Horreee.... it Works.