Update: The SD card I'm using for this is brand new and it is showing corruption sometimes. I think that the SD card is broken but I'm not sure. While I am debugging this I will not give more details to avoid other's damage. In the meantime, if someone is interested and wants to take the risk, please PM me.
------
In summary, this is a method to mount from the PC a sd-ext formatted partition on the SD card inside the phone.
The linux kernel inside Android can potentially understand ext2/3/4, NTFS and other advanced filesystems, but you have to mount them manually to see their data within the phone. That is a hassle, but it can be solved if you are root. However, if you connect your phone to your PC with this setup the alternative filesystems in the external SD card cannot be seen from the PC.
I did not find in Internet how to solve the latter issue, but I figured it out. So here I will explain how:
see the ext4 partition within the phone <-- this is not new!!
see the ext4 partition by connecting the phone to a Linux PC <-- these are the news!!
I do not have a lot of time now to explain the steps in detail. I will update the post later, but basically:
- Using Ubuntu and gparted, I formatted my external SD card with 2 partitions: first one small (32MiB) FAT to satisfy Android and second one big (~16GiB) ext4 where my data remains. This assures that the FAT filesystem cannot grow above the 32MiB limit.
- Then, with Ubuntu and fdisk I deleted (yeah!) the partition table and created a new partition table that says that the FAT filesystem occupies the entire SD card space. This step is the important, it means that the whole card will be exported from the phone when connected to a PC (since the partition table says that there is only one big partition for the whole SD card). However, due to the small size format in the first step the FAT partition cannot grow above the 32MiB mark (where the ext4 filesystem still resides).
- With this setup you can still access the ext4 data (it will not be touched by FAT accesses!!) using mount and the loop device, both from within the phone and from a PC with Linux (if you have another OS, maybe you can use a VM with Linux).
More or less, that's all without the details. I'll update this post later with a step-by-step guide.
Thanks to share that information...
You are welcome. It is a pity that my SD card is not working properly with my phone (it works Ok in the PC). I think that it is a kernel (Froyo) problem so I planned to upgrade to Gingerbread before doing more tests. But right now I have no time...
Note from the Author -
I am moving on to the N5 now and ditching my S3. I will continue to maintain this thread, however - please do PM me if you think that something needs to be changed or updated in this thread as I doubt I will be answering questions within the thread as much. Please don't PM support questions to me. Only PM updates that need to be made in the thread.
It's been a blast!
Regards
Dan
S3 Storage (Data Loss Recovery / Prevention / Info)
This thread is intended to give you an overview of some of the Storage of the S3 from a Data Loss and recovery perspective. It is not intended to cover USB sticks or mods to Swap / Mount other storage. It is solely to cover day-to-day data concerns and give a background to how these things work
Please note, if you have recently swapped between Android 4.1 and 4.2 and cannot find your sdcard data, you need to read [Info] Flashed 4.2? Can't find your /sdcard data?
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Before we get started...
Here are a couple of threads you should get familiar with before posting on XDA.
Forum Rules - use Search before posting
Post Questions or Support queries in Q&A, NOT General
Backround of Android storage (Pre-S3)
Firstly, I think we need to understand how Android worked historically as this will help us to understand how the S3 works now.
A legacy android device, lets pretend the HTC desire on Android 2.2 as this was a standard configuration at the time. It had 2 major partitions (and several minor ones which are irrelevant to this topic). It has the /system partition and the /data partition. These were partitions of the internal NAND flash memory.
/system is where the Android operating system lives. The user could not delete or change anything in here (unless you were rooted). All the software that came with the phone was installed in the /system partition
/data is where all the userdata goes. Whenever you installed an app from the market, it installed to /data/app and it put all it's important data into /data/data. Also any system settings you changed (Wallpaper, ringtones etc) all were stored in /data/data. When you did a factory reset, it wiped /data and everything in it.
Of course, having these 2 partitions was not enough for everyday use. There was no where to store your music, photos, documents etc. /data is just for app data and settings. So this is where /sdcard comes in
/sdcard is the Android mount point for the External SD card in this legacy android device. This means that when you inserted a Micro SD card, Android used /sdcard as it's internal reference to where the card's storage is. The /sdcard was a necessity before you could take photos. Over time, bigger apps started to put other data here. For example, a GPS / Map application would store its apk (application package) in /data/app and store your personal configuration settings in /data/data but it may download and store offline maps somewhere on the /sdcard. In older devices, the internal Memory (NAND Flash) was usually too small to allow much data on it. Many users would have to root to get more storage space or keep uninstalling apps to keep the "low on space" warnings at bay
How the S3 is different
Well, the S3 is substantially different. There are of course SOME similarities. For example, the S3 still has internal NAND Flash Memory. This is often referred to as the eMMC (Embedded MultiMedia Card) - which still contains the /system and /data partitions, used in exactly the same way.
The main difference is /sdcard. The S3 is designed specifically so using an external micro sd card is NOT a necessity. It has a larger NAND Flash internal memory (eMMC) so it can also have an "internal SD card". This is where people start to get confused. The entire internal memory is an eMMC which is essentially an internal SD card, however a partition of that internal memory is /sdcard.
OK, I know - let me explain. /sdcard is a mount point that Android uses to know where to store /sdcard data. But on the S3, instead of storing it on a required external sd card, it points to an internal memory partition. Now here is the clever bit. The /sdcard actually points to /data/media (or /data/media/0 in Android 4.2 onwards). So you continue to have your /data partition, but within /data you have:
/data/app
/data/data
/data/media
The clever thing is that the file system that android uses for /system and /data is a Linux file system called Extended FS. In our case, we use the Ext 4 file system. This is important to understand because these file systems do not work with Windows so an external SD card would usually be Fat 32 file system, or exFAT so we could plug it into our windows computers and read the contents. What Samsung have had to do is use the FUSE file system to allow /sdcard (or /data/media) to exist as a FAT file system within the EXT 4 file system. Clever stuff. But it has it's pro's and cons...
You lost me at file system
All electronic systems that have an operating system and store data use a file system. Think of it in it's simplest form. Imagine a school text book. It has lots of chapters about different things. It has a "Table of Contents" in the first few pages, telling you where each chapter in the book is so if you want to know what page chapter 13 is on, you look in the contents and find the page and go straight there - The alternative is looking through each page individually to find the chapter. Not a quick process.
Well data storage works the same. When you put a file on a hard drive, sdcard, USB stick (or whatever) it is written to a specific location. When it is written to this location, the location is added to the file system. So when you put word.doc onto the drive, The file system is informed of the (very complicated) location of the file. When you tell Windows, Android (or whatever) that you want to open word.doc, the operating system consults the File system and goes to retrieve the data from its true and real (yet very complicated) location on the drive.
There are many file system types still in use today. Usually they are operating system specific. For example, Ext 4 is a Linux file system (and Android by Proxy as it is Linux kernel based) and Windows cannot read Ext file systems. Similarly, exFAT is a Microsoft file system (also used for sdcards on the S3) and cannot be used (easily) on Linux machines. Since most everyday users of the S3 are Windows users, you can see now hopefully why it was important for Sasmung to use "magic" and implement FUSE to allow an exFAT file system to be used for /sdcard, within the Ext 4 partition of /data
Did I REALLY need to know ALL that?
You know what? Probably not but it may go some way to help understand the limitations we will cover later on.
A bit more info for the S3
Obviously, the internal memory supplied with the S3 may not be enough for all users so they added the ability to add expanded memory in the form of the Micro SD card. Android uses the Mount point of /extSdCard now, instead of /sdcard like it used for legacy devices - because /sdcard is already in use elsewhere.
One thing many of you have probably noticed is that with the S3, there is no option to mount the /sdcard or /extSdCard as USB Mass storage on your computer. You must use MTP or PTP.
PTP - Photo Transfer Protocol. When you connect your S3 to your computer using PTP, Your computer sees it as a camera. It will show photos on your "camera" and will set about implementing the default camera options (such as suggesting you import your photos) etc. It won't show documents or other media necessarily.
MTP - Media Transfer Protocol. When you connect your S3 to your computer, it will be seen as a media player. This should allow you full access to all the files on there, including word documents and the such like.
One of the reasons for this is that because /sdcard points to something using the FUSE file system and is not a true partition, it would be difficult to allow it to be used as USB Mass Storage. It may or may not be possible but the biggest advantage of using MTP / PTP is that the computer and the S3 can both access the internal memory at the same time. With USB Mass Storage (UMS) this is quite awkward and can result in errors.
Deleting data
This is partially why we needed to understand a little about file systems. So I could explain to you how data is handled when it is deleted.
As I explained earlier, when you write a file to memory, a corresponding entry is written to the file system to advise the OS where the data is. Sure, you may think you are writing the file to /sdcard/documents/work directory on the internal memory, but in reality these directories or folders do not actually exist at a memory hardware level. The data is written to a block and the file system is informed where that block is, how big the file is, what directory it should appear in to the OS etc. When a file is written the memory, the OS see's the available space go down and the used space go up. All this information comes from the file system.
When you delete the file, the actual data is NOT deleted. It remains where it is on the memory. The block is not overwritten. When the OS is told by the user to delete the file, the File system entry is deleted. This changes the free/used space as the file system is no longer accounting for the data, however the truth is the data still exists. When the next request to write a file to the memory comes from the OS, the file system will think the block where the old data was is empty and will overwrite it.
It is this difference between the file system and reality that allows data to be recovered by external software. if you do not write any data to the memory, external software can scan the memory for data whilst bypassing the file system all together. Ff course the window is small. You only have a very limited time to recover data before the file system allows the data to be overwritten with a new entry.
This is not just true of a deleted file. Even formatting the memory (which is actually just re-creating a new, blank file system) leaves all the data in tact behind-the-scenes and can all still be recovered until you start writing data to the memory. Cool huh?
Wow, all this time I've been stressing, is it really that simple?
Awwww snap! You got me. No it is not that simple. All this PC software, example: Piriform recuva only works on a computer drive. In windows, imagine this is anything with a Drive Letter. C: drive etc.
The only way to get a drive letter on your sdcard is to use USB Mass Storage mode, which as previously discussed - is not possible on your S3 (unless you are rooted, you can mount USB mass storage in custom recovery or use a UMS app from Play). The alternative is to use a card reader on your PC and put the sdcard in it.
There are also apps like Undelete for root users - which again, you guessed it - requires root. So if you're not rooted, it's simplest to use a card reader which can be bought for peanuts.
It's worth mentioning, NONE OF THE ABOVE will work with /sdcard on internal memory. It is not possible to get your data back once deleted from internal memory. Once gone, it's gone forever. You can only restore from /extSdCard (removable, external SDcard)
Phone won't boot, can I get my data back from internal memory?
Let's start by saying, it depends why your phone won't boot. If it's an SDS (Sudden Death Syndrome) type issue, where your internal eMMC (NAND FLash memory) has failed, then no. However, if you believe this is not the case then you can get your /sdcard data using adb BUT you need a custom recovery to be flashed via Odin before you do this. Read [REF] Understanding the basics before rooting your S3
However, if out of curiosity - you do still want to get your data off, using adb , read below:
Pre requisite is having adb "installed" on your windows PC. Download THIS file and follow the instructions in the readme.
You need to observe the following. For android 4.1.x and earlier, /data/media for android 4.2.x and newer, /data/media/0 - I will assume 4.2.2 for this guide,.
1) Boot into recovery, connect usb and go to "mounts and storage". Toggle the "mount data" options to mount these partitions. Tip, when mounted, the option then becomes "unmount data"
2) Open "cmd" in Windows and type the below code, which will copy all your data to a folder called sdcard on your windows desktop
Code:
adb pull /data/media/0/ c:\users\rootsu\desktop\sdcard
Also note, this assumes you have windows vista or newer. Also, it assumes your windows username is rootsu.
That's it, simple.
Display and Digitiser won't work, can I get my data back from internal memory?
You can use adb and a custom recovery to pull data from your /sdcard or even app data from /data/data
Pre requisite is having adb "installed" on your windows PC. Download THIS file and follow the instructions in the readme.
You need to observe the following. For android 4.1.x and earlier, /data/media for android 4.2.x and newer, /data/media/0 - I will assume 4.2.2 for this guide,.
1) Boot into recovery, connect usb and go to "mounts and storage". Toggle the "mount data" options to mount these partitions. Tip, when mounted, the option then becomes "unmount data"
2) Open "cmd" in Windows and type the below code, which will copy all your data to a folder called sdcard on your windows desktop
Code:
adb pull /data/media/0/ c:\users\rootsu\desktop\sdcard
Other things you may want to pull.....
Code:
adb pull /data/data/com.android.providers.telephony/databases/mmssms.db c:\users\rootsu\desktop\sdcard
Code:
adb pull /data/data/com.android.providers.contacts/databases/contacts2.db c:\users\rootsu\desktop\sdcard
Also note, this assumes you have windows vista or newer. Also, it assumes your windows username is rootsu.
That's it, simple.
Data corruption
When data becomes corrupt, there's really not a lot you can do. The file system knows where the data is already. If it's corrupt, you're stuck. Most common causes of corruption are:
1) Dirty unmount of /sdcard. SD card pulled out whilst it is being written to / phone shuts off whilst being written to. SOMETIMES - Plugging the card into a card reader in windows, Windows will ask to fix it and MAY fix it.
2) Fake SD card. This is really the MOST common. Get a card reader ans use:
h2testw.exe for windows to test your card in a card reader. Set it to read the full size of the card, which will take hours but well worth it.
If you get a result like this:
Code:
Warning: Only 63995 of 63996 MByte tested.
The media is likely to be defective.
3.8 GByte OK (8072512 sectors)
58.6 GByte DATA LOST (122989248 sectors)
Details:2 MByte overwritten (4096 sectors)
...Then you have a fake card, that is really 4 GB. I'll explain this.
Commonly, fake cards are reprogrammed to "think" they are high capacity cards, such as 32 GB or 64 GB to defraud buyers out of money. This is common on eBay (Never buy cards from eBay).
When these cards are formatted, the file system also thinks it is this fake capacity. Normally, when a card is full, the file system will report to the OS there is no more space and this prevents additional writes to the card. However, in the case where the card is 4GB and the File system thinks it is 64 GB, the tricked file system doesn't know the card is full. The file system keeps allowing data to be written to the card, over writing the existing data but without replacing the file system entries. The file system thinks data that has been overwritten hasn't been overwritten so when you try to open one of these files, it is essentially "corrupt" or non-existent.
Preventing data loss
Time to wise up guys. It is possible to recover data off your removable media, but internal memory - very unlikely. No apps on your PC or Android will help with deleted data. So you need to backup.
Dropbox - Use dropbox to automatically upload your photos to online storage.
Foldersync - Use FolderSync to upload important sdcard files to your dropbox account, or better yet - got a computer thats always on at home? Set foldersync to schedule a sync over wifi whilst you're asleep.
Other info
Interesting tidbits
Quite an exhaustive reference guide you got here rootSU thanks this will sure come handy for all of us :good:
Cheers
Thank you very much for taking the time to write this. It's a non academic approach to a sum of keywords and all of them are explained in such a manner that it would be almost impossible to misunderstand
Nice!
Nice post!
There are a few other interesting tidbits of info that might be worth mentioning:
- eMMC has an internal micro-controller that runs very specific firmware (and SDS was mainly caused by a bug in that firmware)
- eMMC (just like SSD) has specific writing/erasing limits and commands to deal with that - as a very general idea it can write about 4k at a time but can only erase in much larger blocks - like 64k (at least, but a 16GB model could have a much bigger block); normally on the same erase-size block there is very special list maintained, and based on that list wear leveling is implemented;
- all flash-based memory AGES - there is only a limited amount of erase/writes cycles possible before a point where the info is no longer reliably-stored; in some models that value can be incredibly small! to avoid writing more to some regions than other a mechanism call wear leveling is implemented; that one can have a big impact on both speed and reliability (but really don't expect it to create miracles)
- since it is very important for the speed and reliability of the flash memory to return unused blocks to this internal lists there are special TRIM commands that informs the firmware that the block can be garbage-collected; with an OS that supports TRIM, when a file is erased the blocks are also TRIMmed; this is one extra level that makes recovery basically impossible under normal circumstances
- this does not really mean that things are completely impossible to recover, just that you might need to spend so much on it that recovery would be impractical for any item worth less than 100000 US$ to 1 million US$
EDIT
- also just as with SSD it is not a bad idea to keep a good percentage of the flash memory free - IMHO at least 4GB for 16GB models, 6-8GB for 32GB models - that will improve performance since fragmentation (CLARIFICATION - free-space fragmentation) will grow much slower
- unfortunately there is no program for eMMC similar to smartctl (or any other SMART-data reading program) on normal SATA/IDE/SCSI disks - there seem to be some proprietary commands that are somehow similar but those are generally undocumented.
xclub_101 said:
Nice post!
Click to expand...
Click to collapse
I've put a link to your post in post 1. Where as it's not strictly relevant to my point, it is interesting stuff....
Fragmentation isn't an issue on ssd's. Its an issue on hdd because the head must physically move to another area of the Platter to get the data. That's the slow down. Defrag of a hdd moves all the used blocks (data) together so the actuator doesn't need to move much.
Performance degrades over time on ssds because every write, if data already exists must be erased too. But this hasn't really been an issue so much since TRIM became widely available.
-----------------------
Sent via tapatalk.
I do NOT reply to support queries over PM. Please keep support queries to the Q&A section, so that others may benefit
rootSU said:
...
Fragmentation isn't an issue on ssd's.
...
Click to expand...
Click to collapse
Thank you.
Also sorry for the misunderstanding with my contraction - what I wanted to say was free space fragmentation - that one does matter a lot on solid-state memory because of the garbage collection and some controllers have been famous for having a huge drop in performance with little free space - I will try to also correct that post.
xclub_101 said:
Sorry, I used a misleading contraction - what I wanted to say was free space fragmentation - that one does matter a lot on solid-state memory because of the garbage collection and some controllers have been famous for having a huge drop in performance with little free space - I will try to also correct that post.
Click to expand...
Click to collapse
Yep, it's true about Garbage collection, but TRIM *should* handle this nicely as should "over provisioning" although probably, some cheap SSD's may not over provision.
EDIT> Actually (sorry everyone for off topic) if you're interested in SSD's, these articles are a "fun" read... (I put fun in speech marks as it depends how geeky you are )
http://en.wikipedia.org/wiki/Trim_(computing)
http://www.pcworld.com/article/2038...-ssds-what-makes-these-speedy-drives-hum.html
Update to post 1:
Note from the Author -
I am moving on to the N5 now and ditching my S3. I will continue to maintain this thread, however - please do PM me if you think that something needs to be changed or updated in this thread as I doubt I will be answering questions within the thread as much. Please don't PM support questions to me. Only PM updates that need to be made in the thread.
It's been a blast!
Regards
Dan
Awesome bits of info. This is the game changer. I learned a whole lot just by reading here in XDA. I've only been using Android for a few weeks but thanks to XDA, I've already rooted, installed a bunch of apps and kept my OCD in check.
my device memory has corrupted and when i start recovery mode i get "E: faild to mount /cash (invalid argument) "
Thank Blefish first ,with your contribution to U8800 ,so we users can enjoy the Kitkat.
I have learned that Kitkat will forbid app create file/folder in ext-SD root directory,instead apps can only created their data in /Sdcad/Android.So my Sdcard directory will be tidy.
When I installed Blefish 0516 edition,I found the feature didn't work.
And I found in dir ./storage , ext-SD was mounted as Sdcard0 while the internal storage was mounted as Sdcard1.I think the irregular mounted sequence between ext-Sd and internal storage is the reason why the feature didn't work.
And I didn't found vold.fstab in ./system/etc , so I can't edit it to change the mount seq.
Dear Blefish guru,I am looking for your help~
Jellybean and forward use a different method now. They are not using vold.fstab anymore. Rather, /fstab.qcom is used (in our phone). It cannot be edited manually, a boot.img must be unpacked and repacked in order to do so.
Did you mention that apps can write to /sdcard/ but cannot write to /storage/sdcard0? This ROM does not enforce write permissions as KitKat does by default, so I am not quite sure.
Blefish said:
Jellybean and forward use a different method now. They are not using vold.fstab anymore. Rather, /fstab.qcom is used (in our phone). It cannot be edited manually, a boot.img must be unpacked and repacked in order to do so.
Did you mention that apps can write to /sdcard/ but cannot write to /storage/sdcard0? This ROM does not enforce write permissions as KitKat does by default, so I am not quite sure.
Click to expand...
Click to collapse
Two days after installed 0516,many folders were created by apps in ext-SD root directory(./storage/sdcard0/),while only two folders were found in internal storage(./storage/sdcard1/).And it seems that the two folders were somthing like being created while using recovery or something like that,unlikely being created by apps.
Cause I am using another rom to find whether the seq between internal and ext-SD is normal,yet I found that in android 4.0.4 there is no ./storage,so I cannot know the seq between internal and ext-SD.
zhuld.lord said:
Two days after installed 0516,many folders were created by apps in ext-SD root directory(./storage/sdcard0/),while only two folders were found in internal storage(./storage/sdcard1/).And it seems that the two folders were somthing like being created while using recovery or something like that,unlikely being created by apps.
Cause I am using another rom to find whether the seq between internal and ext-SD is normal,yet I found that in android 4.0.4 there is no ./storage,so I cannot know the seq between internal and ext-SD.
Click to expand...
Click to collapse
In my ROM external SD (/storage/sdcard0) is the primary storage. This means Android creates it's Pictures, Download, Music & other directories there automatically.
/storage/sdcard1 is just a storage, like USB OTG storage would be. It's not in use if sdcard0 is present (some apps can still write to it manually like Camera).
The /storage is another jb/kk feature. I did not go with the older direct mounting to /mnt/ because it's just deprecated. Mine is configured according to this (Nexus One with internal storage added manually): http://source.android.com/devices/tech/storage/config-example.html
Blefish said:
In my ROM external SD (/storage/sdcard0) is the primary storage. This means Android creates it's Pictures, Download, Music & other directories there automatically.
/storage/sdcard1 is just a storage, like USB OTG storage would be. It's not in use if sdcard0 is present (some apps can still write to it manually like Camera).
The /storage is another jb/kk feature. I did not go with the older direct mounting to /mnt/ because it's just deprecated. Mine is configured according to this (Nexus One with internal storage added manually):
Click to expand...
Click to collapse
Thanks for so detailed reply.
But I still don't understand how to enforce write permission.As a perfectionist , I really don't want to see so many folders created by third-party apps in my ext-SD.
If it is convenient for you,could you inform me the way to enable the write permission in your rom? :fingers-crossed:
So this doesn't like fully decrypt the adoptable storage/private data or anything quite like that, instead you are to format your sdcard as fat32 and then I have a script that magisk can run from the /data/adb/service.d/ directory called startup.sh that will mount specific folders that you specify - in my case Android for data & obb files as well as an Emulation folder which I intended to use for RetroArch. The only issue I had with RetroArch though was that it would not see the Emulation folder like other apps would so I ended up also writing a script to update the playlists files so that all the rom files I have can be found.
The point of all this is just to make your apps and things as portable as possible while also being unencrypted. If any of yall like this or have any ideas then please contribute away and fork the project. I will accept PRs if I think it adds useful things. I may also work on setting up a way to watch the /data directory for new apps being installed to the internal storage & maybe automatically relocating them to the sdcard to save space, but I dunno if that'd really work or not, I suspect it would though.
rbreaves/AndroidTV-Decrypted: Specific to Onn 4K, Dynalink & similar Android TV 10+ devices. Rooted, decrypted, mounts fat32 for internal use. (github.com)
I kept having a lot of issues with the adoptable storage sdcards becoming corrupted in a normal Android TV setup and it was extremely irritating because the process to get all the files back on it intact was always too long and drawn out. Now if the sdcard ever becomes corrupt I can quickly and easily restore back to how it was & also make backups of the entire SD card without any issues.
The on boot mounting script is located here btw.
AndroidTV-Decrypted/startup.sh at master · rbreaves/AndroidTV-Decrypted
Specific to Onn 4K, Dynalink & similar Android TV 10+ devices. Rooted, decrypted, mounts fat32 for internal use. - AndroidTV-Decrypted/startup.sh at master · rbreaves/AndroidTV-Decrypted
github.com