Hi,
when i make a full backup with TWRP to my sdcard the result is partly corrupted.
Namely, some of the files, both image and .md5 files have the correct size, but are simply full of 0x00 bytes.
For some image files, there is good data at first and then it switches to 0x00 for the rest of the file.
In one case, even the recovery.log files was zeroed!
Is this a known issue? what could it be generally? i am not terribly familiar with all of this...
thanks.
SM-A310F
recovery.log file: ufile.io/edot0fbk
Related
Hello, I would like to know how I could extract backed up TWRP data on my computer? maybe because the data was large TWRP split it into two files : data.ext4.win000 and data.ext4.win001 (sizes 1.5 and 1.2 GB)
How I got there:
I've been using XenonHD rom for quite a while, and this rom uses koush's superuser app.
I noticed there was a newer version on the market so I installed it; however it appears the one used by the rom developer was modified and therefore had a different package name: I ended up with two superuser apps, one of them asking to update the binaries, which I did.
After that the phone got stuck and rebooted two or three times before being stuck indefinitely at 'updating app' message right after the bootanimation. I tried:
- to wipe dalvik cache but it was still stuck;
- I reflashed the rom + gapps but it was still stuck;
- then I went to the /data/app/ folder (via TWRP) and deleted the superuser app i installed from market, as well as the corresponding data in /data/data, wiped cache and dalvik again: still the same...
-the I decided to make a backup via TWRP : one of the whole system, and one of Data partition only;
- then I realised I couldn't transfer it to my computer in TWRP, so I finally made a factory reset, re-flashed the rom+gapps, and then (it worked) copied my backup to the computer
now; how can I retrieve the data of my apps only, as I guess it is the system data that is messed up?
Of course I tried to restore the Data backup in twrp and it got stuck again the same way....
any help greatly appreciated!
You could try appextractor on the play store
Sent from my Nexus 4 using xda app-developers app
forvrknight said:
You could try appextractor on the play store
Sent from my Nexus 4 using xda app-developers app
Click to expand...
Click to collapse
It didn't work for me...
I finally got my data back by this tricky procedure (there might be an easier way but google+xda was not my friend on this one, I tried ext2explore and ext4explore, linux_reader, Andoird_ICS_JB_ext4_unpacker and maybe some more without success)
I renamed the two files:
data.ext4.win000 renamed as data.ext4.tar.001
data.ext4.win001 renamed as data.ext4.tar.002
Unzipping the first one with 7zip recreates a .tar archive named data.ext4.tar, which is not really a tar archive (extracting it will only give a very few files, for me it only included the image files of my custom bootanimation)
On this file I used AccessData FTK Imager, which would recognise the 'evidence' as a tar file and would allow you to export the data, however this is only the data contained originally in the first file (data.ext4.win000) (although the reconstructed archive was the right size, for me 2.7GB, the part extracted by this software was only 1.5GB)
To recover the second part, simply rename the second file (initially data.ext4.win001) into data.7z and then extract. (this does not work with the first file, all you would get is again a very few files, for me it looked like the content of a single apk file)
All you have to do now is to merge the two data folders and voilà
I could then retrieve my sms mms messages by retrieving the file smsmms.db (and eventually smsmms.db-journal) located in /data/data/com.android.providers.telephony/databases, and putting them on the phone in the same folder
I could also retrieve my phone call logs by retrieving the files contacts2.db (and eventually contacts2.db-journal) located in /data/data/com.android.providers.contacts/databases, and putting them on the phone in the same folder
Hope it helps someone someday
For some reason replacing the data folder of a particular app does not always work, I have to investigate a bit or maybe start fresh with a new rom and try again
asim0 said:
It didn't work for me...
I finally got my data back by this tricky procedure (there might be an easier way but google+xda was not my friend on this one, I tried ext2explore and ext4explore, linux_reader, Andoird_ICS_JB_ext4_unpacker and maybe some more without success)
I renamed the two files:
data.ext4.win000 renamed as data.ext4.tar.001
data.ext4.win001 renamed as data.ext4.tar.002
Unzipping the first one with 7zip recreates a .tar archive named data.ext4.tar, which is not really a tar archive (extracting it will only give a very few files, for me it only included the image files of my custom bootanimation)
On this file I used AccessData FTK Imager, which would recognise the 'evidence' as a tar file and would allow you to export the data, however this is only the data contained originally in the first file (data.ext4.win000) (although the reconstructed archive was the right size, for me 2.7GB, the part extracted by this software was only 1.5GB)
To recover the second part, simply rename the second file (initially data.ext4.win001) into data.7z and then extract. (this does not work with the first file, all you would get is again a very few files, for me it looked like the content of a single apk file)
All you have to do now is to merge the two data folders and voilà
I could then retrieve my sms mms messages by retrieving the file smsmms.db (and eventually smsmms.db-journal) located in /data/data/com.android.providers.telephony/databases, and putting them on the phone in the same folder
I could also retrieve my phone call logs by retrieving the files contacts2.db (and eventually contacts2.db-journal) located in /data/data/com.android.providers.contacts/databases, and putting them on the phone in the same folder
Hope it helps someone someday
For some reason replacing the data folder of a particular app does not always work, I have to investigate a bit or maybe start fresh with a new rom and try again
Click to expand...
Click to collapse
It actually helped me, thanks a lot!
Two Questions
asim0 said:
On this file I used AccessData FTK Imager,
Click to expand...
Click to collapse
This is a 4.6 Gigs professional software. If TWRP can read this type of file there must be a smaller tool available for this.
Can you please let me know of any other toll which would do the same?
asim0 said:
All you have to do now is to merge the two data folders and voilà
Click to expand...
Click to collapse
For the 2nd file do we need to follow the same procedure like 'evidence' and stuff?
Thanks, I am desperate to read some information from TWRP backup and cannot wait 6 days while the 4.6 Gigs of ISO will download.
pi_yush said:
This is a 4.6 Gigs professional software. If TWRP can read this type of file there must be a smaller tool available for this.
Can you please let me know of any other toll which would do the same?
For the 2nd file do we need to follow the same procedure like 'evidence' and stuff?
Thanks, I am desperate to read some information from TWRP backup and cannot wait 6 days while the 4.6 Gigs of ISO will download.
Click to expand...
Click to collapse
These files (at least the ones produced by TWRP2 on my phone) are just standard gzipped tar archives. The tar command has supported this format for at least 20 years. You can extract with a command like this:
Code:
tar xvfz data.ext4.win000
tar xvfz data.ext4.win001
number_thirty_two said:
These files (at least the ones produced by TWRP2 on my phone) are just standard gzipped tar archives. The tar command has supported this format for at least 20 years. You can extract with a command like this:
Code:
tar xvfz data.ext4.win000
tar xvfz data.ext4.win001
Click to expand...
Click to collapse
Thanks number_thirty_two, I copyed thes to a folder I named untar in my home folder of CYGWIN, and extracted my system files perfectly
C:\cygwin\home\carl\untar\
system.ext4.win000
system.ext4.win001
tar xvfz system.ext4.win000
tar xvfz system.ext4.win001
Apologies for resurrecting this, but does anyone know how to extract them if they were compressed as .comp files?
asim0 said:
It didn't work for me...
I finally got my data back by this tricky procedure (there might be an easier way but google+xda was not my friend on this one, I tried ext2explore and ext4explore, linux_reader, Andoird_ICS_JB_ext4_unpacker and maybe some more without success)
I renamed the two files:
data.ext4.win000 renamed as data.ext4.tar.001
data.ext4.win001 renamed as data.ext4.tar.002
Unzipping the first one with 7zip recreates a .tar archive named data.ext4.tar, which is not really a tar archive (extracting it will only give a very few files, for me it only included the image files of my custom bootanimation)
On this file I used AccessData FTK Imager, which would recognise the 'evidence' as a tar file and would allow you to export the data, however this is only the data contained originally in the first file (data.ext4.win000) (although the reconstructed archive was the right size, for me 2.7GB, the part extracted by this software was only 1.5GB)
To recover the second part, simply rename the second file (initially data.ext4.win001) into data.7z and then extract. (this does not work with the first file, all you would get is again a very few files, for me it looked like the content of a single apk file)
All you have to do now is to merge the two data folders and voil
Click to expand...
Click to collapse
I've been scouring the interweb looking for a way to take a stock tar file and convert it to a zip file that will flash with twrp. I know it can be done, but thus far I'm not happy with the processes I've found... how do you guys do it?
The stock TAR files contain partitions in IMG format - an all in one file system, and in this case using EXT4 formatting.
In order to create a ZIP, you need to access the files on the partition, which means finding a way to mount those IMG partitions to copy their contents, or find a piece of software to extract from them w/o mounting. Since they are EXT4, you can probably assume a Windows solution isn't available (EXT4 is Linux). Some utilities may exist to handle EXT4 on Windows, but finding one that knows EXT4 and can extract/mount an IMG file is pretty specific.
EDIT : found a utility for you - http://forum.xda-developers.com/showthread.php?t=2285831
If you can somehow get the files out of the archive, you'll need to find a tutorial on the structure of a flashable ZIP. Generally, its just folders and files but there is also a META-INF folder, scripts to install and set permissions, etc. There are likely threads here on XDA that can detail these steps.
Spitzaf, thanks so much! I was hoping for just such a tool
Hello,
I've got a Q2 smartwatch from IQI (launch q2 or call it the way you want). I've successfully dumped the ROM from it (see attached ZIP file). The ROM is also seperated into bootloader(s) and OS (the OS is the FILE_01_MTK). When I run binwalk on this file, I get alot of false positives. All the image files are real, but there are alot (to the point I need to pipe the command with more) of LZMA compressed data (see http://pastebin.com/rdfBRLxe)
When I try to extract the LZMA archive starting from the first bit of reported data, it simply says that it is a corrupted lzma archive.
Also, there is a reported Linux EXT filesystem at 0x8D3803 (9254915), but it is reported as unclean and can't be mounted when I extract it (see file extfs in ZIP).
Can someone help me figure out how I can extract all this data in order to make some changes to the OS? I have experience in hacking other firmware, but never saw so much false positives.
Thanks,
Acrilex123
I have used LGUP and selected all partitions from DUMP. Is it enough if something goes wrong, do I have everything I need? It must have specific extension?
Yes you have everything you need. Partitions dont need a file extension you can add .img to the file names if you want, which will make them image files. They can all be flashed via fastboot or twrp using console commands just the way they are.
While trying to modify my KW88 Pro watch to run AsteroidOS I took the advice of the program and clicked "Format All + Download" without fully understanding what it does. It deletes all partitions. The install then failed with some random errors but I had a brick. I managed to recover it with KW88 Pro firmware (KW88Pro_V1.5_NHH_B_20190114) I found on discourse.fullandroidwatch.com (don't know how legit it is but at least it works now).
While this guide is how to fix the "PMT changed for the ROM, it must be downloaded" error, the above is relevant. You must find firmware for your watch that has the default partition layout if you did not take a backup before formatting or if you don't know the partition layout. In a nutshell you get the PMT error when the partition layout of the scatter file does not match the partition layout of the device, and the stock firmware + scatter file is needed as a reference point. My error:
https://imgur.com/aFZKfOo
Let's compare the start addresses of those 3 partitions to the stock firmware:
https://imgur.com/rV3XBt2
Boot and Logo partitions are the same, but User Data starts at a different address.
If we compare the contents of both the scatter files we see the same thing:
https://imgur.com/OYw10d3
https://imgur.com/DNQ1Ztw
In a nutshell what you need to do is clone the partition layout (start address variables + offset variable) into the new scatter file, but maintain all other variables of the scatter file. It should also be mentioned that there are other partitions in the scatter file that I had from AsteroidOS that did not match the physical layout of the firmware I used, and even though I was not flashing them I still had to edit the scatter file.
Afterwards using these settings I flashed successfully:
https://imgur.com/TdVRDEV
I also want to note that the End Addresses will not match even with the correct partition layout because different firmware has different sizes on disk, despite the space allocated for the partitions. In theory if you want you could reclaim unused space by tweaking the original scatter file and then the custom scatter file, but it may also impact your future updates if the partition is too small.
Hope I made someone's day easier, best of luck
Can you explain the getting the partition start and offset address part?
Because I think im going to brick my phone really soon if I can't make this auto-generated scatter I got from internet work to flash my phone back to stock rom.
The scatter itself already has some bugs with userdata partition (Too large)
kutiz21 said:
Can you explain the getting the partition start and offset address part?
Because I think im going to brick my phone really soon if I can't make this auto-generated scatter I got from internet work to flash my phone back to stock rom.
The scatter itself already has some bugs with userdata partition (Too large)
Click to expand...
Click to collapse
Sure thing. The scatter file which is for stock firmware vs. the file for new firmware will not allocate the space on disk based on what partitions are actually, physically on it. The new scatter file for custom firmware will be for what the devs had on their device, but not what you actually have on yours (different hardware versions, android releases, w.e.). What you need to do is take the existing partitions of your device, the start addresses for them, add the size of that partition to get the end address and using that information supply that data to the new scatter file. You COULD in theory also change the partition sizes but that is more prone to error so I did not recommend that. So what I did would visually look like this, the | are partition start / end markers
Original scatter file:
|xxxxxxxxxxx||xxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxx|
vs new scatter file:
|xxxxxxxxxxx||xxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxx|
vs the patched scatter file:
|xxxxxxxxxxx||xxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxxxx....................||xxxxxxxxxxxxxxxxxxxxxxx....................................................||xxxxxxxxxx........|
Tl;Dr An offset is basically a way of saying address + data/space (until next segment at next address).
Intersting
printf() said:
Sure thing. The scatter file which is for stock firmware vs. the file for new firmware will not allocate the space on disk based on what partitions are actually, physically on it. The new scatter file for custom firmware will be for what the devs had on their device, but not what you actually have on yours (different hardware versions, android releases, w.e.). What you need to do is take the existing partitions of your device, the start addresses for them, add the size of that partition to get the end address and using that information supply that data to the new scatter file. You COULD in theory also change the partition sizes but that is more prone to error so I did not recommend that. So what I did would visually look like this, the | are partition start / end markers
Original scatter file:
|xxxxxxxxxxx||xxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxx|
vs new scatter file:
|xxxxxxxxxxx||xxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxx|
vs the patched scatter file:
|xxxxxxxxxxx||xxxxxxxxxxx||xxxxxxxxxxxxxxxxxxxxxxxxxxx||xxxxxxxxxxxxxxx....................||xxxxxxxxxxxxxxxxxxxxxxx....................................................||xxxxxxxxxx........|
Tl;Dr An offset is basically a way of saying address + data/space (until next segment at next address).
Click to expand...
Click to collapse
I used Smartphone flash tool version 3 and 5 and I have managed to use my own image to flash numerous phones, encrypted with a key or with no keys (Auth file).
Usually, I would use Google's method of flashing back rom that is using a zip file contains bunch of flashable system images and let fastboot app itself flash it.
I dumped my images using dd tool and img2simg (Full blown raw ext4 file, not sparsed.)
Because fastboot likes sparse images rather than raw ones and flashable sparse images can be easily flashed with SP tool normally (It can make phones bootable but will more likely making phone unbootable most of the time)
I do aware that I need to adjust the address thing. But normally, using the start address based on the scatter. SP app can automatically calculates how large the image and it can prevents you from flashing because it is too large.
But this particular phone won't flash even if it is the stock rom. Which dumped and got its scatter file generated by some sketchy "miracle box".
My best guess is that the scatter file was tied to specific emmc chip and it is not for my phone's emmc so there might be some difference there.
Because when I use the REAL flash scatter file from actual OEM (In this case, Nokia 1, European Variant). It can make my APAC counterpart can be flashed normally, even with my custom sparsed or raw image (These 2 variants were kinda different, technically.)
I would like to point out that newer SP flash tool app tends to hate to co-operate with scatter scripts so that could be a reason (I used older build of version 5 and it only gets the userdata bug, but it does lags the app when the bug "PMT changed for the ROM, it must be downloaded" show up. Because it was a bug and it hadn't fixed in that build, which is fixed in the latest build of SP app when I attempt to flash)
And flashing extracted "stock" rom from flash tool and flash it back over fastboot tool tends to break system OTA and make my phone on bootloop (I highly doubted it was the raw image thing, because of the free space it can allocated. I flashed it raw back then I think because i don't know what happened yet, keep in mind that stock rom images shipped for SP flash tool were most likely in raw state, not compressed like sparse images)
And SP flash tool also hates raw dd images too (too big?????)
So like. What do I need to fix the script? What program that can calculate the exact offset for it? I still be able to flash the stock rom with fastboot fine. But for "unforeseen consequences" that makes it unbootable, I still need it.
Say the scatter is not reliable or doesn't exist. How do I use the readback function and then use some aged apps like MTK droid tool to generate actual offsets and possibly generates scatter for me (Most likely will not work because my phone's Android is too new for it). I used to work with Android 4.2.2 so it's really easy back then.
kutiz21 said:
So like. What do I need to fix the script? What program that can calculate the exact offset for it? I still be able to flash the stock rom with fastboot fine. But for "unforeseen consequences" that makes it unbootable, I still need it.
Say the scatter is not reliable or doesn't exist. How do I use the readback function and then use some aged apps like MTK droid tool to generate actual offsets and possibly generates scatter for me (Most likely will not work because my phone's Android is too new for it). I used to work with Android 4.2.2 so it's really easy back then.
Click to expand...
Click to collapse
I'm not sure how to get the data when no original scatter file exists, I was fortunate enough to find a reference for my stock OS. The other scatter file was for AsteroidOS. I think the easiest way to explain what I did would be to share the scatter files. Effectively start address + partition size should not be larger than the start address of the next partition.
Stock scatter file:
https://pastebin.com/tfFDCPfT
Original AsteroidOS scatter file:
https://pastebin.com/0z7DtL9u
Patched AsteroidOS scatter file:
https://pastebin.com/iiYZTzBt
Take note that in my original post screenshots, AstreroidOS was only flashing 3 partitions (corresponding to the is_download: true/false field of the scatter file). That told me that those were the partitions / boundaries that need to match, the other partitions were irrelevant (for the purpose of flashing anyway). AsteroidOS itself is contained inside the userdata partition, effectively leaving other ~20 partitions as "lower layers" or unused.
printf() said:
I'm not sure how to get the data when no original scatter file exists, I was fortunate enough to find a reference for my stock OS. The other scatter file was for AsteroidOS. I think the easiest way to explain what I did would be to share the scatter files. Effectively start address + partition size should not be larger than the start address of the next partition.
Stock scatter file:
https://pastebin.com/tfFDCPfT
Original AsteroidOS scatter file:
https://pastebin.com/0z7DtL9u
Patched AsteroidOS scatter file:
https://pastebin.com/iiYZTzBt
Take note that in my original post screenshots, AstreroidOS was only flashing 3 partitions (corresponding to the is_download: true/false field of the scatter file). That told me that those were the partitions / boundaries that need to match, the other partitions were irrelevant (for the purpose of flashing anyway). AsteroidOS itself is contained inside the userdata partition, effectively leaving other ~20 partitions as "lower layers" or unused.
Click to expand...
Click to collapse
So my insights were true. It does automatically calculates the free space of a partition to whether allow you to flash or not. But I tried fiddling with it and it doesn't work. I might try to individually comment out those partitions as unflashable and try it out.
One more thing is that I used to had this error. But it will be gone when I explicitly referencing SP flash tool the preloader partition image (flashing it or not doesn't matter) alongside with all of my custom images but they must be fully loaded in the app.
You will not be able to flash if it can't fully loaded all partitions that has the property "is_download: true". Maybe that should allow me to do the flash.
Any more thoughts on " boundary_check and is_reserved"? Because I used to poke with these and managed to flash my other phone successfully.
I don't have the phone to test right now but more food for thought I guess.
Cheers!
kutiz21 said:
Any more thoughts on " boundary_check and is_reserved"? Because I used to poke with these and managed to flash my other phone successfully.
I don't have the phone to test right now but more food for thought I guess.
Cheers!
Click to expand...
Click to collapse
Not sure what is_reserved does, but boundary_check probably validates the scatter file vs. the layout on the disk. If you don't do the check though you are risking overwriting a partition which might be needed for the lower layer functionality - keystore, firmware, etc.