Database Users

[Q] CMR/Nandroid backup FAT32 filesize limitation

Hi!
So I'm running civato's FLEXREAPER-R6 ICS 4.0.3 ROM on my Iconia A500 after using timmyDean's "root-3.2.1-V4.7z" method to root my stock HC 3.2.1 ROM. I had ClockworkMod Recovery v5.0.2.3 (rev 1.3.4 by thor2002ro) and was unable to do a full Nandroid backup so I upgraded to the latest ClockworkMod Recovery v5.5.0.x (rev 1.7 by thor2002ro) but I'm still having the same issue.
When I try to backup data I get the error "Error making a backup image of /data!" and then when I look at the sdcard in \clockworkmod\backup\YYYY-MM-DD.HH.MM.SS I see "data.ext4.tar" and the size is 4,294,967,296 bytes), which is exactly the 4GB filesize limit on a FAT32 partition. Looking inside the data.ext4.tar file, I see data/app and all the .APK installer files for apps installed on my Iconia plus a few .ZIP files as well. ES File Explorer won't show me how big /data/app is but DiskUsage says it's 4.7GB. From what I'm reading Android needs a copy of all the APK user-installed app installation files of in /data/app so that it can restore the default data/configuration when requested (under Settings/Apps/Manage Apps/Delete Data & Cache). So...how can I get a successful Nandroid backup if I have more than 4GB of apps installed? Can the data portion be broken up into <4GB chunks? Or can Nandroid/Recovery be updated to support exFAT or NTFS? Or would it be safe to exclude /data/app from the backups?
Since I've been unable to do the Recovery/Nandroid backup of "data", I've been doing a custom backup of everything except for data and then backing up all my installed apps and settings from inside the OS with Titanium Backup. But with this method, a catastrophic failure would still require me to use Recovery to restore the OS and then use Titanium to restore app my apps.
What's the solution?

You can try doing several backups, in other words, do a nandroid backup of /data separate from the other back ups of /system /boot /recovery and other paritions. Then when you have to restore, first restore the backup with your system partitions and then restore your data partition afterwards. You are probably backing up to many things to have a nandroid file that large anyway. When you do a full data wipe it does not wipe the data on the internal sd card only the data on the data partition.

cruise350 said:
You can try doing several backups, in other words, do a nandroid backup of /data separate from the other back ups of /system /boot /recovery and other paritions. Then when you have to restore, first restore the backup with your system partitions and then restore your data partition afterwards. You are probably backing up to many things to have a nandroid file that large anyway. When you do a full data wipe it does not wipe the data on the internal sd card only the data on the data partition.
Click to expand...
Click to collapse
Hey cruise350! I have an Evo 3D as well. I'm running Virus's "Eternity 3.0 r193 ROM" with the Stock theme on my Evo 3D, how about you? Anyway, I have tried doing separate Nandroid backups. I can backup everything but "data" and that will complete just fine. I don't see any way to break up the "data" backup into multiple smaller backups. But as I said, if I backup everything else with Nandroid, I feel pretty confident that I can restore all my apps/data with Titanium.
Would it be safe to try to move /data/app temporarily somewhere else (such as /mnt/sdcard), do the Nandroid data backup, and then move it back?
It seems to me that doing full wipe (system, data, cache, dalvik cache, superwipe, etc) would wipe out /data and everything underneath (including /data/apps) but that it would get recreated automatically when I restore everything with TB, right?

The way to get rid of that problem once and for all is to format your SD card as EXT4... You won't be able to read it with non-Linux (or BSD) OSes after that, though. Mounting via USB should still work, as this doesn't access the SD filesystem directly...

I would try changing recoveries then, I am using RA recovery and it has a compress backup feature. It takes about 45 minutes to do a nandroid when compressing but it cuts the size of the backup significantly. I bet RA will solve your problem though. And, I am running Steelrom on my Evo 3d. It's really stable, has some good tweaks, and the battery life is incredible.

haag498 said:
The way to get rid of that problem once and for all is to format your SD card as EXT4... You won't be able to read it with non-Linux (or BSD) OSes after that, though. Mounting via USB should still work, as this doesn't access the SD filesystem directly...
Click to expand...
Click to collapse
Thanks, haag498. That's an interesting idea. So any Android OS or Recovery would be able to read/write to an EXT4-fomatted SD card? It would work in both my Iconia A500 and my Evo 3D both in Recovery and inside the OS? That might be a good option. I see that EXT4 supports very large filesizes so that problem would go away. What tool would you recommend for formatting the SD card with that filesystem? Can it be done from inside Android?
Not being able to mount the filesystem natively in Windows might present occasional inconveniences but it shouldn't be a major problem as long as the Android devices can present the storage as USB Mass Storage mode. That being said, I have some issues with the Iconia when accessing the storage over USB. Namely, when I try to move files and folder off the device or delete them from Windows when attached via USB, the task often won't complete. I've worked around the problem by just doing internal moves, deletes and renames from inside Android (using ASTRO or ES File Explorer) and, if necessary, mounting the SD card directly in my Windows PC, which won't be possible anymore.

cruise350 said:
I would try changing recoveries then, I am using RA recovery and it has a compress backup feature. It takes about 45 minutes to do a nandroid when compressing but it cuts the size of the backup significantly. I bet RA will solve your problem though. And, I am running Steelrom on my Evo 3d. It's really stable, has some good tweaks, and the battery life is incredible.
Click to expand...
Click to collapse
Okay...so if I wanted to install RA Recovery instead, would I flash that from inside CMR? I have "Acer Recovery Installer" installed as well but I believe the version of CMR I'm using is newer than what that tool would have been able to install. So, what's the procedure to switch to RA from CMR? Any other features of RA that you prefer?

JesseAaronSafir said:
Okay...so if I wanted to install RA Recovery instead, would I flash that from inside CMR? I have "Acer Recovery Installer" installed as well but I believe the version of CMR I'm using is newer than what that tool would have been able to install. So, what's the procedure to switch to RA from CMR? Any other features of RA that you prefer?
Click to expand...
Click to collapse
You can flash from within CWM. If you look in the RA threads, you will find some flashable versions. I use 3.15, as I have been lazy to update to 3.16.
No wipes needed. Just install zip from SD
http://forum.xda-developers.com/showpost.php?p=22392691&postcount=104

JesseAaronSafir said:
Thanks, haag498. That's an interesting idea. So any Android OS or Recovery would be able to read/write to an EXT4-fomatted SD card?
Click to expand...
Click to collapse
They should, if they're using a reasonably up-to-date Linux kernel, as EXT4 is built in for these... If you want to be absolutely sure, use EXT2/3 (they're pretty much identical from a user point of view), which still support large files and all, but are somewhat slower (and they can't use more than 4TB, which shouldn't be an issue anyway...).
JesseAaronSafir said:
It would work in both my Iconia A500 and my Evo 3D both in Recovery and inside the OS?
Click to expand...
Click to collapse
Read above.
JesseAaronSafir said:
That might be a good option. I see that EXT4 supports very large filesizes so that problem would go away. What tool would you recommend for formatting the SD card with that filesystem? Can it be done from inside Android?
Click to expand...
Click to collapse
Formatting the SD card can be done from any Linux live CD (Ubuntu, Knoppix, ...) or from within Android (if you're rooted and have a chroot Linux) using parted (or cfdisk) or some GUI tool like gparted... Internally, they'll all use mkfs.ext4, which is anything but user-friendly, though...
JesseAaronSafir said:
Not being able to mount the filesystem natively in Windows might present occasional inconveniences but it shouldn't be a major problem as long as the Android devices can present the storage as USB Mass Storage mode. That being said, I have some issues with the Iconia when accessing the storage over USB. Namely, when I try to move files and folder off the device or delete them from Windows when attached via USB, the task often won't complete. I've worked around the problem by just doing internal moves, deletes and renames from inside Android (using ASTRO or ES File Explorer) and, if necessary, mounting the SD card directly in my Windows PC, which won't be possible anymore.
Click to expand...
Click to collapse
That's actually a known problem... I usually just use a NAS for data exchange, as it has plenty of space and good speed, even on WiFi. Another way to do it would be a USB drive (2.5'' hdds tend to need more power for spin-up than they get from the internal port, though). Also, as I use Linux on all my computers, mounting EXT4 partitions is no problem for me ...

CWM making backups that are very small (@15mb)?

Using the newest CWM recovery touch, I'm trying to make a backup before jumping to AOKP (hooray!) But.... My backups are tiny; about 15mb. I'm not sure how big they should be (the one I made immediately after rooting was 880mb) but I'm certain this is too small. Anyone have any idea what I'm doing wrong? Thanks in advance.

If you've used an older version of CWM, I'm sure you're used to ~1GB backups. In 6.0 and above, backups are handled differently. A good portion of backups are put into these "blob" files and then a smaller backup is made for each specific backup. The theory is that if you have multiple backups, they can pull data from the blob files instead of duplicating things that haven't changed. Whatever you do, DO NOT delete these blob files, doing so will corrupt your backup.

meijin3 said:
If you've used an older version of CWM, I'm sure you're used to ~1GB backups. In 6.0 and above, backups are handled differently. A good portion of backups are put into these "blob" files and then a smaller backup is made for each specific backup. The theory is that if you have multiple backups, they can pull data from the blob files instead of duplicating things that haven't changed. Whatever you do, DO NOT delete these blob files, doing so will corrupt your backup.
Click to expand...
Click to collapse
Thank you! I was so confused by this. So if I want to transfer my backup to my laptop HDD, I now need to copy the entire Clockworkmod folder? (and this will have all of the files necessary for a full backup?)

slack04 said:
So if I want to transfer my backup to my laptop HDD, I now need to copy the entire Clockworkmod folder? (and this will have all of the files necessary for a full backup?)
Click to expand...
Click to collapse
Unfortunately yes, the new blobs system means it will take a longer time to make a copy of your backup, it's horrible over Wi-Fi. I just didn't bother with it and went with TWRP instead.

[Q] Questions regarding the two NST rooted with the same process

I rooted two nooks, and made tar backups of their partitions (I also got the dd image just in case). Out of curiosity, and with the goal of keeping minimal backup and increasing the partition for side-loaded contents to maximum, I tried to compare the contents of each partition. Since I followed the same process for rooting (Touch-Formatter v2, 1.2.1 update, NookManager, NTGAppsAttack - but before booting the Nook I got the backup), I guessed quite a lot of them are the same, and found some interesting results.
1. Boot partition is nearly the same except uRamdisk. I inspected the contents of the two uRamdisk files using bootutil by Renate, and they are identical.
-> Why are they different, and can one replace the other?
2. As we all know, rom partitions are different, but it looks only a few of them can meaningfully affect the operation. Anyway, it's small and I decided to leave them untouched and keep two separate copies.
-> What is the BCB file by the way? It just has zeroes inside. Is it automatically created if not there?
-> Some files in the devconf directory seem to be modified during the normal process or firmware update, notably BootCnt (four zero bytes), Bq275020Dffs (12 in the rombackup.zip, 13 after 1.2.1 update). What are these? Any idea?
3. The factory partition, I want to bust it (empty it up and resize it to the minimum), and let me know if I'm on a dangerous path. The idea is that I don't need rombackup.zip because I can revive the rom partition with my own tar backup if something bad happens, and it's out of date anyway after 1.2.1 update (some files in the rom partition are modified). Also, with Touch-Formatter and CWM, I wouldn't need the factory.zip file.
-> What are the files in the "touch" directory? One of my nook has them, but the other doesn't. Looking at the data inside they must be related to the display or touch screen. Maybe byproduct of calibration?
-> Can I use 1.2.1 update file with CWM instead of using Touch-Formatter, bringing it to a new fresh 1.2.1 Nook? According to this post, it seems possible.
4. The system partitions are identical, as expected. but with CWM recovery, we wouldn't need a backup of it, right?
5. The cache partition is way too big. I know the firmware update uses this space (when I resized it to something like 64MB, 1.2.1 update didn't work. I needed to increase it to something like 128MB to make it work). However, for normal operation, we surely don't need it that big.
-> How small can it be? I know it depends on individual's usage patterns... but in my case, I mostly use Nook for reading side-loaded contents. I've gone down to 32MB, but I guess that's still big.
-> Do we really need the cache partition? Can we just symlink it to somewhere in the data partition?

Out of curiosity, I just deleted BCB and BootCnt in the rom partition, and rebooted. First it said "Install Failed", a screen I have never seen before on Nook. So I looked into the rom partition and found that BCB file is recreated, but not BootCnt. On the subsequent boot, it said "Installing Rom", and then quickly rebooted. Now it's back to work. So, I guess these two files are essential for normal operation. Again, this time I deleted all files in the factory partition and also deleted BootCnt. Now, it tries to do the "Installing Rom" thing, but fell back to "Install Failed" screen. I opened up the rom partition again and I saw only BCB and BootCnt files, and none else. Nook surely formatted the rom partition first before trying to recreate it.
So I wonderfully bricked my Nook, and thought this is a good time to test if the rom partition backup works. I mounted the rom partition, untarred the backup, and rebooted. There we go, the Bronte Sisters are back. So the conclusion is that
1. When the BCB file's missing, it's simply recreated after a failed boot.
2. When the BootCnt file's missing, Nook thinks the rom partition is corrupted and tries to recreate it using rombackup.zip in the factory partition. I think this may have some side effects because firmware updates only change the files in the rom partition, leaving rombackup.zip untouched. So you will go back to the old rom partition after the built-in rom recovery.
3. The best rom recovery, I think, is using your manual backup of the rom partition. And maybe updating the rombackup.zip with a new one too?

BootCnt is a 32 bit little-endian count of the number of failed boots.
Once it hits eight your Nook will boot into the recovery image uRecImg, uRecRam.
You could also echo about anything to that file to make it arithmetically greater than 8.
Code:
echo 000 > /rom/devconf/BootCnt
That is 0x0a303030 > 8
Normally this is a B&N thing that asks you about factory restore.
If you replaced those two files it could be Clockwork Mod Recovery.

[GUIDE] Decrypt your OnePlus One

So you thought that encrypting your OnePlus One was a good idea? So did I. I had so many issues after having encrypted my One that I simply wanted to get rid of it. After a lot of searching, I didn't come across a simple guide to decrypt it. And here I am
First thing first. Did you know that TWRP doesn't do a _full_ nandroid backup? It backs up everything _except_ /data/media. That was basically the reason why I'm writing this.
What you need:
An unlocked bootloader
TWRP recovery (I was using 2.8.6.1 but any version might just work)
An USB-OTG dongle with a memory stick that is large enough to hold all the data
A full battery
Here are the steps:
Go into TWRP recovery and make a backup. Be sure to select the usb_otg as the target location for your backup
Go into TWRPs terminal and make a backup of your /data/media folder. When starting the terminal, you will be asked in which folder you want to start. Navigate to "/data/media" and ckicl "select". I used tar to achieve this. Simply type in "tar cvpzf /usb_otg/datamedia.gz ." (Note: there is a dot a the end!) and it will create an archive with everything in it including permissions.
Now wipe your device. *scary* I know... Still in TWRP go to "Wipe", "Format Data" and type in "yes". This won't just wipe your data but also wipe the encryption
Since our device is basically empty, we just have to restore eveything. Let's start by restoring the nandroid backup
After the nandroid backup, let's restore /data/media. For this go back into TWRP terminal. Starting folder will be "/data/media" again. Type in "tar xvpzf /usb_otg/datamedia.gz ." (Note: there is a dot a the end!) to restore everything.
There you go... OnePlus One decrypted!

Simple and clear guide. Just what I was looking for. A couple of questions before I begin:
File size limit. Most USB drives are FAT32 (4GB file limit) so how do you split the tarball into <4GB files?
Will TWRP (or Android or whatever recognizes filesystems) recognize exFAT, NTFS or other filesystems that don't have file size limitations?
If you do split the tarball into smaller sizes, how do you restore them?
Thanks again for the guide and info

Advice

I need some help with my Elsa. I soft bricked it about a 9 days ago while I was installing gapps on top of lineage OS 14.1. I had rooted using Cow mid January , 17.
Using fat I resized, formated, wiped ----system, data, cache--- partitions. Then I formated Dev/Art in F2FS which resulted in some very interesting stuff. Now that I've gotten TWRP back on it ( or helped it revive ) through fastboot.
File system looks good.
/system
file system ext4
size 5416MB
free 5405MB
used 10MB
backup size 10MB
/data
file system ext4
size 52528MB
free 5245MB
used 0MB
backup size 0MB
/cache
file system ext4
size 0MB
free 0MB
used 0MB
backup size 0MB
Is there any cmds, coding, prompts.... etc.... i can run in the twrp terminal to gather more information to the condition of the device?
No stock. No roms. Nothing except Android I think
A clean blank state?
The device Had full SElinux and MTP this whole time it was bricked

twidledee said:
I need some help with my Elsa. I soft bricked it about a 9 days ago while I was installing gapps on top of lineage OS 14.1. I had rooted using Cow mid January , 17.
Using fat I resized, formated, wiped ----system, data, cache--- partitions. Then I formated Dev/Art in F2FS which resulted in some very interesting stuff. Now that I've gotten TWRP back on it ( or helped it revive ) through fastboot.
File system looks good.
/system
file system ext4
size 5416MB
free 5405MB
used 10MB
backup size 10MB
/data
file system ext4
size 52528MB
free 5245MB
used 0MB
backup size 0MB
/cache
file system ext4
size 0MB
free 0MB
used 0MB
backup size 0MB
Is there any cmds, coding, prompts.... etc.... i can run in the twrp terminal to gather more information to the condition of the device?
No stock. No roms. Nothing except Android I think
A clean blank state?
The device Had full SElinux and MTP this whole time it was bricked
Click to expand...
Click to collapse
if you have twrp, the flash a rom for your device via twrp and let it boot.
typically will reboot 2x and take 10-15 min from start to finish 1st boot.
if you have twrp you just need to flash a rom,
make sure you do a full wipe, and a format data prior to installing your rom.
Please stop re-posting new threads, your making this harder for any dev to help as you've now made 3 posts for the 1 device you have, and the 1 issue.

Team DevDigitel said:
if you have twrp, the flash a rom for your device via twrp and let it boot.
typically will reboot 2x and take 10-15 min from start to finish 1st boot.
if you have twrp you just need to flash a rom,
make sure you do a full wipe, and a format data prior to installing your rom.
Please stop re-posting new threads, your making this harder for any dev to help as you've now made 3 posts for the 1 device you have, and the 1 issue.
Click to expand...
Click to collapse
I'm sorry.
It's just been hard. I'm reading and trying to do my homework on this issue from LG UP to boot img. Recovery img. And system img. Having issues with adb seeing device to Windows drivers not being correct.
Now that I've got twrp functioning again on the device along with adb... I've got the KDZ file which I have extracted to produce a system.img file. I have no boot or recovery img though. I've put all extracted files into my platform tools folder. I don't know what to do now. I don't want to mess this up. Help me get this correct
Sent from my zeroltetmo using XDA Labs

twidledee said:
I'm sorry.
It's just been hard. I'm reading and trying to do my homework on this issue from LG UP to boot img. Recovery img. And system img. Having issues with adb seeing device to Windows drivers not being correct.
Now that I've got twrp functioning again on the device along with adb... I've got the KDZ file which I have extracted to produce a system.img file. I have no boot or recovery img though. I've put all extracted files into my platform tools folder. I don't know what to do now. I don't want to mess this up. Help me get this correct
Sent from my zeroltetmo using XDA Labs
Click to expand...
Click to collapse
if you have twrp you dont need a system.img recovery.img or a boot.img
you just need to go to https://forum.xda-developers.com/v20/development/rom-natf-1-00-test-5-debloated-stock-t3509750
download the rom,
place on sd card and install it using twrp.

Team DevDigitel said:
if you have twrp you dont need a system.img recovery.img or a boot.img
you just need to go to https://forum.xda-developers.com/v20/development/rom-natf-1-00-test-5-debloated-stock-t3509750
download the rom,
place on sd card and install it using twrp.
Click to expand...
Click to collapse
Is sbin the internal storage I need to sideload to for the rom and gapp package I want to install?
Oh sdcard... I see it.
adb sideload /sdcard filename.whatever?
Sent from my zeroltetmo using XDA Labs

twidledee said:
Is sbin the internal storage I need to sideload to for the rom and gapp package I want to install?
Sent from my zeroltetmo using XDA Labs
Click to expand...
Click to collapse
You can set the zips anywhere in internal.. But id recommend putting them on a sd card as they wont accidently get erased.
With his rom you dont need gaps.
Only aosp roms or aokp roms
Just choose wipe and do it 3 times
Then do advanced options in wipe and do format data
Once done go to install and choose the location where the rom was saved. Again a sd card eliminates side load. Mounting ywrp with mtp to move stuff etc. But if you have to use internal. Go to mount and mount mtp stoarge and use pc to drag zip to phone.
Should just need to install the rom and reboot. The link i sent you is for the stock rom you started with.. And some mod options. Just get it working with it and get the fine details done later..
But you just need to find the file under install and install the full zip. If its aroma it will let you personalize rom.
Just install the recommended zip from link and profit.

Team DevDigitel said:
You can set the zips anywhere in internal.. But id recommend putting them on a sd card as they wont accidently get erased.
With his rom you dont need gaps.
Only aosp roms or aokp roms
Just choose wipe and do it 3 times
Then do advanced options in wipe and do format data
Once done go to install and choose the location where the rom was saved. Again a sd card eliminates side load. Mounting ywrp with mtp to move stuff etc. But if you have to use internal. Go to mount and mount mtp stoarge and use pc to drag zip to phone.
Should just need to install the rom and reboot. The link i sent you is for the stock rom you started with.. And some mod options. Just get it working with it and get the fine details done later..
But you just need to find the file under install and install the full zip. If its aroma it will let you personalize rom.
Just install the recommended zip from link and profit.
Click to expand...
Click to collapse
I appreciate your very informative and knowledgeable advice. A+brother
Sent from my zeroltetmo using XDA Labs