Database Users

Booting Android from sd card on Dream/G1

Hi all,
Here are my findings about how to boot android from an SD card, useful for example to test development Android builds without messing your phone. This procedure was inspired by legacy GNU/Linux boot process and then should work on most hardware with a flashable recovery.
########################################
I can't stress enough the fact that this procedure is targeted to experienced user. A good knowledge of linux/android booting process is more than required. This procedure is not meant to be useful to most people given its limitations (no recovery mode, rebuild of boot.img required)
########################################
I won't propose a step by step tutorial as it's better to understand how to do it and adapt the procedure to each need.
Two modifications are required:
1. Declare to mount sd card partitions instead of internal flash volumes. This has to be done in the init.rc script.
For example:
mount yaffs2 [email protected] /system
mount yaffs2 [email protected] /system ro remount
with
mount ext2 /dev/block/mmcblk0p6 /system
2. For some reasons, in the boot process, sd card partitions generally show up later than the mount actions, so it is generally necessary to add a timer in init binary to give to to the kernel to detect sd card partitions before mounting them (adding a sleep(5) after the "A N D R O I D" text in init.c is enough).
Build your special boot.img embedding those two modifications and flash it on the recovery volume of your phone. Now, booting into recovery will launch the system on your sd.
Regards,

I would like to report success on booting the G1 from a system located on sd card (cyanogenmod 4.2.7.1). I will describe the procedure step by steps when every thing is okay.
There is just an issue that you might have encountered during cyanogenmod ROM cooking, all wireless connexions (wifi, BT, GSM) do not work. The same system on flash if okay. I made a diff on boot log messages and the only notable difference is this error :
E/MemoryHeapBase( 83): error opening /dev/pmem_gpu0: Permission denied
E/MemoryHeapBase( 83): error opening /dev/pmem_gpu1: Permission denied
E/MemoryHeapBase( 83): error opening /dev/hw3d: Permission denied
I tried to chmod 0555 /dev/pmem* with no success.... Any idea ?

Although I finally managed to have a usable system, this work is still experimental. Please fell free to test and report success, failures here so I can improve the thing.
Let's see how we can boot a copy of cyanogen ROM 4.2.7.1 located on sd card. The main purpose is to test future ROMs (eclair ?) without messing up your phone...
DISCLAIMER : This procedure is targeted to experienced user. I am not responsible if you loose your data, your phone or your Mom !
Prerequisites are :
- adb and fastboot operationnal on your host computer
- Boot image file boot.img :
http://www28.zippyshare.com/v/19654421/file.html
- files data.cpio.gz, system.cpio.gz (data.cpio.gz and system.cpio.gz are unmodified images of a fresh install of cyanogen ROM that fit the modified boot.img) from :
http://www15.zippyshare.com/v/57489279/file.html
http://www15.zippyshare.com/v/13077582/file.html
1.
Create 3 partitions on sd card. The first one (vfat) is to store your music, videos etc. The second and the third will hold /data and /system. Result shoul look like this :
# fdisk /dev/block/mmcblk0
Command (m for help): p
Disk /dev/block/mmcblk0: 1977 MB, 1977614336 bytes
64 heads, 63 sectors/track, 957 cylinders
Units = cylinders of 4032 * 512 = 2064384 bytes
Device Boot Start End Blocks Id System
/dev/block/mmcblk0p1 1 864 1741792+ c Win95 FAT32 (LBA)
/dev/block/mmcblk0p2 865 903 78624 83 Linux
/dev/block/mmcblk0p3 904 957 108864 83 Linux
Command (m for help):
2.
Copy data.cpio and system.cpio on the first partition of your sd card.
3.
Connect to your device with adb :
adb shell
4.
Mount mmcblk0p2 on temporary folder and extract user.cpio.gz archive :
mkdir /dev/tmp
mount /dev/block/mmcblk0p2 /dev/tmp
cd /dev/tmp
gunzip -c /sdcard/data.cpio.gz | cpio -i
cd /
umount /dev/tmp
5.
Mount mmcblk0p3 on temporary folder and extract system.cpio.gz archive :
mount /dev/block/mmcblk0p3 /dev/tmp
cd /dev/tmp
gunzip -c /sdcard/system.cpio.gz | cpio -i
cd /
umount /dev/tmp
6.
Reboot the phone in bootloader :
adb reboot bootloader
7.
Boot the phone with custom boot image :
unzip boot.img.zip
fastboot boot boot.img
Reports are welcome !

I haven't tried this, but great idea. Well done.
Maybe a nice additional idea would be a boot-menu style idea. Eg, detect OS on mem card and display a menu of "Internal" or "SD Card"?

I'm on it but as it will imply flashing people's phone, I need more testing and suggestions from experienced hackers. I only own my G1 for 5 days...

I acytually just started a thread about this in the Dream development subforum!
Some suggestions:
-Use a recovery image to boot the ROM on the sd card (look at modifying the CM recovery image) if this is possible. This way, you can boot from sd card when you want to, by booting into recovery image, without interfering with the flashed rom at all. No worries that flashin a new rom will require a reflash of your bootloader. Also, you can use the ROM installed in flash after a reboot without any hassle, which would be very useful when testing a ROM from the SD if it doesn't work very well and you need a working phone.
-Have the /data and /system-partition as folders on the 4th partition on the SD card, or as logical volumes on a primary one. Why?
Well, many people have their sd-cards set up with fat, extX, swap in that order. The partition layout you have described here simply isn't compatible with that, and will require a separate SD card just for this testing (which everyone might not have).
I must say, I think this is a GREAT idea! I so often want to test a few ROMS, but often they don't get the test-time they deserve because I need to swicth back to my working environment for job/uni the day after. This would be a great way to test a ROM thoroughly. And also it would be the best way to give a ROM a quick testdrive. switchroming back and forth is, for all its simplicity, hassle.

Using the recovery is the only way I found at the moment to boot from the sd card and was about to extend cm recovery with a dedicated menu Nevertheless, there will be a limitation with that: it will not be possible to use a different kernel for the system on sd as it will have to use the kernel of the recovery... Anyway, many custom roms around here use cm kernel (even those I saw with eclair) so it is not so problematic I think (tested Eugene373 AOSP20 yesterday). Anyway, it is possible to adapt the recovery with a test kernel...
I think I found a workaround for sd partitionning scheme pb, using bind mounts but I have not tested already. I will work on it this WE.

If I remember correctly you can use the command "reboot recovery" from the recovery shell to reboot again into recovery. It could mean that it is possible to choose what (kernel) to boot after the reboot. Even cyanogenmod has made quite a few changes to the kernel since the recovery image came out, and I think it would not be a very good solution to use the same kernel as the recovery image for all ROMs loaded via SD. (Especially the Hero ones won't work at all, I'm afraid.
It could also be possible maybe, to tweak the built-in bootloader into booting form either SD-card or from the internal flash? It already has the possibility to boot different things on different keypresses (home for recovery mode, and camera for fastboot). I have (again) no idea of its capabilities for reading anything off of the sd card, though.

I get your point naguz . I am not satisfied either with the solution of using the recovery kernel to boot the system on sd. I found that it adds quite much complexity to the init process: I tweaked the recovery executable to add an entry to boot from sd but I faced troubles in services startup and pre-init definitions. I think that the solution of using the recovery menu to choose to boot from sd have to be abandoned as it will require heavy additional changes to the init.rc scripts of the second system and will break its advanced features.
The ideal solution (as you suggested) would be to tweak the bootloader to boot natively from the sd card but unfortunately, we do not have the sources of the SPL (tell me if I am wrong) so it is definitely not possible.
The remaining solution is to use the recovery partition to flash the boot image of the second system. I works well, just press the home key and the second system boots ! The drawback is that you do not have recovery any more... Personnaly, I don't find that so problematic as I is still possible to boot a recovery image with fastboot when needed, so I think I will stick to this solution. I somebody have another solution, I am ok to investigate...

Aha, it makes sense that booting the sd directly from recovery mode would mess something up. I would think some of the same problems would be faced when booting the kernel from the recovery partiotion, doesn't it look like a different device to the kernel? Well, if it works, it works.
Regarding the source of the SPL, I have no idea, but I know hyakuro (a (former?) user here) has released a modified one. Trying to get in touch with him
As for the latest method: Is the recovery partiotion big enough to hold both the recovery image AND the kernel? If so, one could maybe have both. Maybe make a new "recovery image" that can either boot from sd or boot recovery image? Just throwing out ideas here.
Personally I don't see the big problem with not having a recovery image, as I would (in a dual-boot scenario) already have another, working install on the flash that I could use if the one booted from sd wasn't good enough. Re-flashing the recovery image could also be done from the working ROM in flash, for those without the SDK tools.
I think, however, that quite a few people will object to not having a recovery image.
Btw, was your latest working test done with one (4th) partition on the sd card for loading the ROM from SD? If so, new instructions please. I'd like to give it a try.

I think that it is technically impossible to boot directly from the sd, even with the sources of the SPL as drivers are required to drive the sd that can not be included in a SPL. It is the same issue on PC with PCMCIA network cards for instance. IPL+SPL has to be seen more or less like a simple BIOS...
The recovery partition size is not a matter her. The problem is the lack of control over what has to be booted as the only action we can make in the SPL is the HOME key to choose a regular boot or a recovery boot (the system on SD here). I think I have a good knowledge now of the boot process and I think we can not got further than that, despitely...
I am now trying to have /data and /system on the same partition, mount them on /mnt and bind mount each directory on /system and /data but with no succes . investigating....

Okay, bind mounting was a dumb idea of mine. The solution is to create an extended partition with 2 logical drives in it, so partitions are:
[1-3] FAT and/or EXTX and/or swap as needed (I have personnaly just one partition here)
4 extended
5 ext2 for /data
6 ext2 for /system
Now its time to bake a boot.img with the kernel of the system on the sd card + the ramdisk. I do not explain how to do that, google knows :
1.
In the ramdisk image, put my modified init program attached hereater
2.
In the init.rc script, change the lines that mount /data and /system :
mount yaffs2 [email protected] /system
mount yaffs2 [email protected] /system ro remount
with
mount ext2 /dev/block/mmcblk0p6 /system
and
mount yaffs2 [email protected] /data nosuid nodev
with
mount ext2 /dev/block/mmcblk0p5 /data
Flash boot.img on the recovery and reboot in recovery... That's all.

Jahrome,
Trying the idea on the HTC has a subtle difference. So I'm curious what change you put in the /init process.
Lets say the image one wants to use is the 'eclair' branch. This branch uses vold, so the block device, such as '/dev/block/mmcblk0p#', does not appear to be created until the vold service is started, which is after the init.rc mount operations. So the mount proposed does not work.
Curious what changes you made to the init process that might give me some ideas of what might be the simplest change. It seems that there is a relatively easy solution, that I don't yet see.
Ideas (or additional questions to clarrify) appreciated...

Hi Dale !
On which HTC are you trying the trick ?
First your trouble is not related to eclair as I successfully booted eclair on my G1 with this trick. You need to add a timeout in init so as it waits for the kernel to detect sd card partitions (need to recompile init, or use my recompiled init).
Vold has nothing to do at this point
Hope this helps

Init Delay for SDisk
Thanks for the reply. That's what I needed, it seems so ungraceful.
I thought vold was enabling something to make the kernel's discovery of the SD partitions. It is "by coincidence" that the log of the vold occured shortly before the adding of the block-device event. My apologies for not recognizing the coincidence of the log entries.
I had hoped the solution was a bit more "graceful", though I can't say what that would be or why a sleep is "ungraceful". I would like a "mount retry for n-seconds" option in the init.rc, that would be slick. Nonetheless I will add in a sleep-spin-check for a few seconds.
For the record, we are discussing the change to:
android-source: .../system/core/init/init.c
code-routine: main()
code-location: Somewhere after the "A N D R O I D" text
code-change: add in a sleep spin-check of some sort
I appreciate the comment that all we need is just a sleep. I think this completes the thread?!

Most interesting...
This is something akin to how I am using my company's Android solution for the Beagle Board...
In that environment, the entire kernel and rootfs are located on a SD/MMC card. The environment variables and boot script are stored in the NAND chip
through a "setenv" command. The U-boot monitor on the hardware defaults to run the boot script unless there is any user interaction. This image can then be converted into a typical distribution that can be flashed and ran without the SD card...
I have wondered about the implementation of this approach on my G1, but I have not had time to test it out. It is good to see that there are others that are interested in this as well...
I will be following this thread and will try to help in any way if I can... meaning if I can get some free time...
If you are interested in our Beagle Board solution, it is open source and can be found by a simple Google search using "Android rowboat"...
L8R...

very easy fix.
rename the /init to /init.android (or whatever you like)
create an init script. have it prompt on boot too boot from sd or internal. symlink the init.rc from /system. That way you only need one boot.img for multiple builds. We've been doing this a really long time on the android on vogue project. My development phone is simple, I hold down the menu button if I want to boot from the sdcard otherwise it boots from the internal.
Oh and don't worry about this.
jahrome said:
log messages and the only notable difference is this error :
E/MemoryHeapBase( 83): error opening /dev/pmem_gpu0: Permission denied
E/MemoryHeapBase( 83): error opening /dev/pmem_gpu1: Permission denied
E/MemoryHeapBase( 83): error opening /dev/hw3d: Permission denied
Click to expand...
Click to collapse
That's just that you don't have HW3D enabled in your kernel. It's important to note that if you're trying to use multiple builds that eclair with 3d and donut use different kernels.

Exactly the vogue and kaiser have been doing this for ages. If you want you can also boot ext2 and squashfs images. It's pretty simple stuff.

Anyone have a backup of the files in post #3? I've been trying this solution with files of my own creation (as in: my own cooked boot.img, and data/system files) but it doesn't seem to get past the G1 boot screen (as in: blank screen.) I've also tried the recovery boot solution however it gives me the same reaction. It leads me to believe the problem lies in my boot file.

ok anyone interested in creating a simple step by step manual? and can someone post the files again please?

[Q] Can't mount /dev/block/mmcblk0p8

I would like some help please.
Have bricked my a500 during attempt to falsh vach leaked ICS.
Switchme was used before clearing data and I forgot to delete a profile. This is I think the cause of my problem.
I cannot proceed with any ROM flashing until data partition is cleared.
I am using RA-ICONIA-3.15 recovery kernel.
I can mount and format CACHE SYSTEM FLEXROM partitions.
Attempting to mount data this is the message
E:Can't mount /dev/block/mmcblk0p8 (File exists)
Now my poor A500 will not get past 'acer'screen when trying to boot.
Question 1: How can I restore access to the DATA partition?
Question 2: How do I wipe my internal storage and start over?
I have a nand backup before this started on external SD card. Uncertain if I should restore this.
Thanks in advance.

DrPlumEU said:
I would like some help please.
Have bricked my a500 during attempt to falsh vach leaked ICS.
Switchme was used before clearing data and I forgot to delete a profile. This is I think the cause of my problem.
I cannot proceed with any ROM flashing until data partition is cleared.
I am using RA-ICONIA-3.15 recovery kernel.
I can mount and format CACHE SYSTEM FLEXROM partitions.
Attempting to mount data this is the message
E:Can't mount /dev/block/mmcblk0p8 (File exists)
Now my poor A500 will not get past 'acer'screen when trying to boot.
Question 1: How can I restore access to the DATA partition?
Question 2: How do I wipe my internal storage and start over?
I have a nand backup before this started on external SD card. Uncertain if I should restore this.
Thanks in advance.
Click to expand...
Click to collapse
try this
If you are using Thor's CWM. And his recovery only;
Here are the steps to completely wipe all internal data. And give it a clean internal system (according to Thor)
Note: Optional Formatting.
If you really want to remove EVERYTHING in Internal Memory, at this point, go to Backup and Restore. Select "Toggle backup and restore of internal storage (/data/media). Enable this setting by selecting it. By default, it is disabled. Then proceed to the formatting steps. This will format the Entire Internal Memory when you proceed.
Note, this will format EVERYTHING in internal memory (not ext sd card)
Select Go Back, Goto Mounts and Storage
1. Format System
2. Format Cache
3. Format Dalvik
4. Format Flexrom
After you reboot, I believe it resets back to the defaults.
try to install the rom with the normal procedures.
this what i used and it worked great no problems
note** i am just fowarding info that was passed to me. and this will erase you backups
---------------------------------------------------------------------------------------------
IF you are running RA Revamped Recovery, then you may want to contact Drellisdee at his thread;
http://forum.xda-developers.com/show....php?t=1459821
He knows the procedures to do the same.it's a tricky process, and he knows more about his recovery than I do.

Thank you for the prompt response. Already tried your recipe with no luck.
I need access to the DATA partition and do not know how to go about that. Once have access to the internal disk, would like to know how to completely wipe this if that what it takes.
Thanks
PS: If I truly have bricked the device it is still in 12mo warranty, so will contact Acer care centre if no tricks work.
PPS: Maybe I now have excuse needed to upgrade to Tegra 3 device. The Acer Iconia A700 looks tempting

i think i have seen something like that when fixing mine but did not touch it....as im not a pro, hopefully somone with a litle more in depth knowledge sees this and can help.
sorry

Have you tried restoring your nandroid backup? This has recovered the error for me on my phone before when the /data partition became corrupt. Worth a shot.

Sadly, restoration of the nandroid backup was not successful.
I have notice that I pressed wrong volume button when attempting the hard reset. Will try the correct procedure later after work. Hopefully a proper hard reset then restore with one of Vache stock roms will have me up and running.
UPDATE: Sadly hard reset fails to load. My poor a500 needs resuscitation at acer clinic

I had the exact same problem, although I caused the problem in a different manner. You need to be able to use adb to get it going again. If you are not familiar with adb, do a search and you can find plenty of help. That's what I had to do. Unfortunately, my skills with adb are pretty much non-existent, so I can't really help you with that part.
Once you get adb going on your computer and it recognizes your A500, follow these steps:
You will need to boot into my recovery and open adb shell.
This will cause loss of all data contained on /data which will lose your /data/media.
From adb shell:
mount (This is to check data isn't mounted. If mounted use "umount data")
mke2fs -j -b 4096 /dev/block/mmcblk0p8
tune2fs -O extents,uninit_bg,dir_index -C 1 /dev/block/mmcblk0p8
e2fsck -fy /dev/block/mmcblk0p8
After this is done your file system on /data will be fixed. You can install the ROM of your choice. Good luck!
(These instructions were originally provided to me by drellisdee)

Thanks for the answer on your previous post. I fixed my /data mount.
Love to unbrick my acer for the 50th time
Did not have this error tho, before I tried to convert the /data mount to ext4

DrPlumEU said:
I would like some help please.
Have bricked my a500 during attempt to falsh vach leaked ICS.
Switchme was used before clearing data and I forgot to delete a profile. This is I think the cause of my problem.
I cannot proceed with any ROM flashing until data partition is cleared.
I am using RA-ICONIA-3.15 recovery kernel.
I can mount and format CACHE SYSTEM FLEXROM partitions.
Attempting to mount data this is the message
E:Can't mount /dev/block/mmcblk0p8 (File exists)
Now my poor A500 will not get past 'acer'screen when trying to boot.
Question 1: How can I restore access to the DATA partition?
Question 2: How do I wipe my internal storage and start over?
I have a nand backup before this started on external SD card. Uncertain if I should restore this.
Thanks in advance.
Click to expand...
Click to collapse
If you switch the A500 off and whilst switching it on hold the volume up button and start toggling the rotation lock on and off it should clear the internal memory completely. Then flash an update.zip from a MicroSD. Otherwise use the Iconia software update tool that's floating around in APX mode. Make sure you have your UID for this

Ping192 said:
Did not have this error tho, before I tried to convert the /data mount to ext4
Click to expand...
Click to collapse
That's exactly what caused my problem. Glad you are working again.

Resolved by great customer support...
To close this thread I wish to congratulate the Acer customer support team in Plymouth who have restored my Acer Iconia A500 to working order in under a week.
It is upgraded OTA to Android 3.2.1.
As still in warranty, no charge for the failure of the hard-drive partitioning!
Great work. Good customer service.

thanks lovable people in this thread
thanks lovable people in this thread you solved my problem . especially Dr. McKnight (23rd February 2012, 03:48 PM) back in the beginning of the thread. His solution using thor's recovery was enough to solve my problem. Wonderful and thank you
/Mikkel

spoupard said:
From adb shell:
mount (This is to check data isn't mounted. If mounted use "umount data")
mke2fs -j -b 4096 /dev/block/mmcblk0p8
tune2fs -O extents,uninit_bg,dir_index -C 1 /dev/block/mmcblk0p8
e2fsck -fy /dev/block/mmcblk0p8
Click to expand...
Click to collapse
When I do the "mke2fs -j -b 4096 /dev/block/mmcblk0p8" command from adb's shell, it takes forever (I let it run for hours). Something I was thinking about, could I simply "rm -rf /dev/block/mmcblk0p8" first, and then run the above commands? Also, should I not add the "-t ext4" option to the mke2fs command?

kerijan2003 said:
When I do the "mke2fs -j -b 4096 /dev/block/mmcblk0p8" command from adb's shell, it takes forever (I let it run for hours). Something I was thinking about, could I simply "rm -rf /dev/block/mmcblk0p8" first, and then run the above commands? Also, should I not add the "-t ext4" option to the mke2fs command?
Click to expand...
Click to collapse
Try in fastboot mode execute:
fastboot -w

Help missing 13gb data - nexus 7

Okay so i was just doing a bit of file management. I removed a 1.4gb psp iso i had on my nexus 7 and then i checked how much space i had left using storage option in settings. I checked it and it said i had 25gb free when before i removed my file i had only 13gb free. Tried rebooting device and then used es file explorer to see all files. Opened es file explorer and then noticed everything was gone. Only stock items were folders were left eg Android, Download etc. All my msuic and some game data was gone. Opened up asphalt 7 to see if it would still work but it doesnt anymore. Tried opening real racing 3 And it asks me to redownload game data. My widgets are still in homescreen and work and some of the other apps i have still work as well eg plague inc, subway surfer and most of the other ones.
Any idea on how this happened and how i can recover my files? I have a bugsense file as well that was left on my device.

As to recovery of lost files your options are not good. (And if we're talking a non-rooted device, the odds are approximately equal to 0%) Recovery in ext4 filesystems is technically much more challenging than in simple filesystems such as FAT. And this pessimistic outlook presumes that the filesystem is healthy/clean. If the reason for the problem occurring in the first place was a corrupted filesystem, then the odds go from simply bad to pathetically poor.
Sorry, dude... got any Nandroid or TiBu backups stored on your PC?
If you had Putin's top-secret files on your N7 and the CIA got hold of it, the first thing a forensic analyst would do is try to take a raw (block) device dump off of the "cold" device. (If you are still running the N7 with the regular OS the /data partition is being continually written to, and this further reduces the chances of file recovery every second the device is booted).
In the case of an analyst with less resources, this might mean using a custom recovery boot to get the raw device copy; unfortunately, the /data partition is huge - nearly 30 GB - so you would have to mount a extN filesystem via OTG... and doing so thus precludes using adb, so you would need to use a recovery with a touch interface and command-line entry (e.g. TWRP)
# mkdir /mnt/myOTGdisk
# mount -t ext2 -o rw /dev/sda1 /mnt/myOTGdisk/
# dd bs=8196 if=/dev/block/platform/sdhci-tegra.3/by-name/UDA of=/mnt/myOTGdisk/userdata.img
Doing such a thing would allow you to examine that huge image file with forensic file recovery tools from a PC (probably running Linux) as in principle you captured the entire ext4 filesystem.
The thing is, efforts spent in file recovery should be proportional to the value of the files being recovered. I'm not sure if your saved gaming history rises to that occasion. For sure the dude at the CIA won't want to help you with that.
As to the source of your troubles, it's hard to say. With TWRP booted, you can run the "e2fsck" program to see if the /data ext4 filesystem is corrupted, e.g.
# mount | grep /data ( see which mmcblk0 partition is /data, on grouper it is mmcblk0p9 )
# umount /sdcard
# umount /data
# e2fsck -f -n /dev/block/platform/sdhci-tegra.3/by-name/UDA
(For the last command above, you might need to use the block device name /dev/block/mmcblk0p?? instead of the UDA symlink )
If the above command shows that you have a corrupted /data filesystem, I would re-initialize that filesystem ( "fastboot format userdata" ) - note this wipes all userdata including the psuedo-SD card.
Finally, I should point out that some type of hardware failure might have occurred somewhere in that huge 30 GB partition - if that is the case then there will be problems down the road again. If that is the case, the only way to detect this will be a write test which nearly fills that partition, followed by a filesystem sanity check as shown above.
Probably that would need to be done in the recovery rather than in the normal OS, as a nearly full /data filesystem will probably wedge the device.
Phew, I've said enough.
Good luck

I never tire of reading your posts, bftb0, ("...the odds are approximately equal to 0%")...genius.
But don't the CIA have access to Cray, 'Kasparov' DeepBlue beating SuperComputers that could make mincemeat out of the kind of thing your alluding to... in less time than it takes to flash a ROM... or have I been watching too many James Bond movies?
Vaguely rhetorical question - think I already know the answer...
Still... what a great post.
Rgrds,
Ged.
---------- Post added at 01:22 AM ---------- Previous post was at 12:50 AM ----------
Hi, leont1280...
You could try running this ...
Disk Usage - http://play.google.com/store/apps/details?id=com.google.android.diskusage&hl=en
It gives a graphical 'map' or overview of your storage, and you can visually see where everything is (or should be), great for tracking down missing stuff... but as bftb0 has mentioned, it doesn't look promising.
Rgrds,
Ged.

Use astro file manager u can check it out
Sent from my Nexus 7 using Tapatalk 2

Right bftb0 i did what you said in twrp and i recieved the following summary information.
/dev/block/mmcblk0p10: 28133/1835008 files (3.2% non-contiguous) , 1571673/733977
Any idea on what that means?
Also under the mount option in Twrp i can Unmount and mount the System, Data and its Cache. However i cant Mount the SD Card. Should that be a concern?

bftb0 said:
As to recovery of lost files your options are not good. (And if we're talking a non-rooted device, the odds are approximately equal to 0%) Recovery in ext4 filesystems is technically much more challenging than in simple filesystems such as FAT. And this pessimistic outlook presumes that the filesystem is healthy/clean. If the reason for the problem occurring in the first place was a corrupted filesystem, then the odds go from simply bad to pathetically poor.
Sorry, dude... got any Nandroid or TiBu backups stored on your PC?
If you had Putin's top-secret files on your N7 and the CIA got hold of it, the first thing a forensic analyst would do is try to take a raw (block) device dump off of the "cold" device. (If you are still running the N7 with the regular OS the /data partition is being continually written to, and this further reduces the chances of file recovery every second the device is booted).
In the case of an analyst with less resources, this might mean using a custom recovery boot to get the raw device copy; unfortunately, the /data partition is huge - nearly 30 GB - so you would have to mount a extN filesystem via OTG... and doing so thus precludes using adb, so you would need to use a recovery with a touch interface and command-line entry (e.g. TWRP)
# mkdir /mnt/myOTGdisk
# mount -t ext2 -o rw /dev/sda1 /mnt/myOTGdisk/
# dd bs=8196 if=/dev/block/platform/sdhci-tegra.3/by-name/UDA of=/mnt/myOTGdisk/userdata.img
Doing such a thing would allow you to examine that huge image file with forensic file recovery tools from a PC (probably running Linux) as in principle you captured the entire ext4 filesystem.
The thing is, efforts spent in file recovery should be proportional to the value of the files being recovered. I'm not sure if your saved gaming history rises to that occasion. For sure the dude at the CIA won't want to help you with that.
As to the source of your troubles, it's hard to say. With TWRP booted, you can run the "e2fsck" program to see if the /data ext4 filesystem is corrupted, e.g.
# mount | grep /data ( see which mmcblk0 partition is /data, on grouper it is mmcblk0p9 )
# umount /sdcard
# umount /data
# e2fsck -f -n /dev/block/platform/sdhci-tegra.3/by-name/UDA
(For the last command above, you might need to use the block device name /dev/block/mmcblk0p?? instead of the UDA symlink )
If the above command shows that you have a corrupted /data filesystem, I would re-initialize that filesystem ( "fastboot format userdata" ) - note this wipes all userdata including the psuedo-SD card.
Finally, I should point out that some type of hardware failure might have occurred somewhere in that huge 30 GB partition - if that is the case then there will be problems down the road again. If that is the case, the only way to detect this will be a write test which nearly fills that partition, followed by a filesystem sanity check as shown above.
Probably that would need to be done in the recovery rather than in the normal OS, as a nearly full /data filesystem will probably wedge the device.
Phew, I've said enough.
Good luck
Click to expand...
Click to collapse
Right bftb0 i did what you said in twrp and i recieved the following summary information.
/dev/block/mmcblk0p10: 28133/1835008 files (3.2% non-contiguous) , 1571673/733977
Any idea on what that means?
Also under the mount option in Twrp i can Unmount and mount the System, Data and its Cache. However i cant Mount the SD Card. Should that be a concern?

leont1280 said:
Right bftb0 i did what you said in twrp and i recieved the following summary information.
/dev/block/mmcblk0p10: 28133/1835008 files (3.2% non-contiguous) , 1571673/733977
Click to expand...
Click to collapse
Hmmm, mmcblk0p10 - you must have a tilapia (3G N7) device, yes?
If you had any filesystem errors, that e2fsck run would have produced copious reams of output. If a filesystem is clean, it produces only 5 or 6 lines of summary output.
leont1280 said:
Any idea on what that means?
Also under the mount option in Twrp i can Unmount and mount the System, Data and its Cache. However i cant Mount the SD Card. Should that be a concern?
Click to expand...
Click to collapse
I've seen the same business with TWRP and the /sdcard mount - I wouldn't worry about it. (It is not a "normal" mount in the sense of extN or FAT device partition mount - it behaves sort of like a strange symlink where the target directory and descendants all appear to have different file ownership and permissions than what exist in the true (underlying) filesystem. No doubt this is all performed in the kernel... I don't know whether a command-line invocation of "mount" can create this mount point, or whether a specific syscall/ ioctl is needed)
But back to your N7 - the lack of any errors in the filesystem check is good news, but also suggests that your files didn't disappear through a hardware failure. Are you sure you didn't fat-finger things when using the file manager? (I suppose it is possible that the file manager has a bug...)
I didn't look into what tools are available for extN forensic/recovery work. I can guess that the effort would be non-trivial, though.

bftb0 said:
Hmmm, mmcblk0p10 - you must have a tilapia (3G N7) device, yes?
If you had any filesystem errors, that e2fsck run would have produced copious reams of output. If a filesystem is clean, it produces only 5 or 6 lines of summary output.
I've seen the same business with TWRP and the /sdcard mount - I wouldn't worry about it. (It is not a "normal" mount in the sense of extN or FAT device partition mount - it behaves sort of like a strange symlink where the target directory and descendants all appear to have different file ownership and permissions than what exist in the true (underlying) filesystem. No doubt this is all performed in the kernel... I don't know whether a command-line invocation of "mount" can create this mount point, or whether a specific syscall/ ioctl is needed)
But back to your N7 - the lack of any errors in the filesystem check is good news, but also suggests that your files didn't disappear through a hardware failure. Are you sure you didn't fat-finger things when using the file manager? (I suppose it is possible that the file manager has a bug...)
I didn't look into what tools are available for extN forensic/recovery work. I can guess that the effort would be non-trivial, though.
Click to expand...
Click to collapse
Im pretty sure i didnt accidently delete it myself. When i was doing file management i was using my laptop with the N7 3g connected to it via MTP. Once i deleted the iso file the N7 started acting strange. I did notice a bit of lag that was usually out lf the ordinary and when i checked available space left it increased to 25gb rather than saying 13gb

aaah, having the same problem here, i was cleaning my n7 using my laptop, found a strange folder on my sd card, looked inside and it has a back up of some of my deleted files! interesting! i deleted the folder, then i started cheking other folders, like my ebooks, audio books and etc, but all my folders were empty!
so i disconnected my tablet, and after reconnecting, bam, all files gone.

[Q] Questions regarding the two NST rooted with the same process

I rooted two nooks, and made tar backups of their partitions (I also got the dd image just in case). Out of curiosity, and with the goal of keeping minimal backup and increasing the partition for side-loaded contents to maximum, I tried to compare the contents of each partition. Since I followed the same process for rooting (Touch-Formatter v2, 1.2.1 update, NookManager, NTGAppsAttack - but before booting the Nook I got the backup), I guessed quite a lot of them are the same, and found some interesting results.
1. Boot partition is nearly the same except uRamdisk. I inspected the contents of the two uRamdisk files using bootutil by Renate, and they are identical.
-> Why are they different, and can one replace the other?
2. As we all know, rom partitions are different, but it looks only a few of them can meaningfully affect the operation. Anyway, it's small and I decided to leave them untouched and keep two separate copies.
-> What is the BCB file by the way? It just has zeroes inside. Is it automatically created if not there?
-> Some files in the devconf directory seem to be modified during the normal process or firmware update, notably BootCnt (four zero bytes), Bq275020Dffs (12 in the rombackup.zip, 13 after 1.2.1 update). What are these? Any idea?
3. The factory partition, I want to bust it (empty it up and resize it to the minimum), and let me know if I'm on a dangerous path. The idea is that I don't need rombackup.zip because I can revive the rom partition with my own tar backup if something bad happens, and it's out of date anyway after 1.2.1 update (some files in the rom partition are modified). Also, with Touch-Formatter and CWM, I wouldn't need the factory.zip file.
-> What are the files in the "touch" directory? One of my nook has them, but the other doesn't. Looking at the data inside they must be related to the display or touch screen. Maybe byproduct of calibration?
-> Can I use 1.2.1 update file with CWM instead of using Touch-Formatter, bringing it to a new fresh 1.2.1 Nook? According to this post, it seems possible.
4. The system partitions are identical, as expected. but with CWM recovery, we wouldn't need a backup of it, right?
5. The cache partition is way too big. I know the firmware update uses this space (when I resized it to something like 64MB, 1.2.1 update didn't work. I needed to increase it to something like 128MB to make it work). However, for normal operation, we surely don't need it that big.
-> How small can it be? I know it depends on individual's usage patterns... but in my case, I mostly use Nook for reading side-loaded contents. I've gone down to 32MB, but I guess that's still big.
-> Do we really need the cache partition? Can we just symlink it to somewhere in the data partition?

Out of curiosity, I just deleted BCB and BootCnt in the rom partition, and rebooted. First it said "Install Failed", a screen I have never seen before on Nook. So I looked into the rom partition and found that BCB file is recreated, but not BootCnt. On the subsequent boot, it said "Installing Rom", and then quickly rebooted. Now it's back to work. So, I guess these two files are essential for normal operation. Again, this time I deleted all files in the factory partition and also deleted BootCnt. Now, it tries to do the "Installing Rom" thing, but fell back to "Install Failed" screen. I opened up the rom partition again and I saw only BCB and BootCnt files, and none else. Nook surely formatted the rom partition first before trying to recreate it.
So I wonderfully bricked my Nook, and thought this is a good time to test if the rom partition backup works. I mounted the rom partition, untarred the backup, and rebooted. There we go, the Bronte Sisters are back. So the conclusion is that
1. When the BCB file's missing, it's simply recreated after a failed boot.
2. When the BootCnt file's missing, Nook thinks the rom partition is corrupted and tries to recreate it using rombackup.zip in the factory partition. I think this may have some side effects because firmware updates only change the files in the rom partition, leaving rombackup.zip untouched. So you will go back to the old rom partition after the built-in rom recovery.
3. The best rom recovery, I think, is using your manual backup of the rom partition. And maybe updating the rombackup.zip with a new one too?

BootCnt is a 32 bit little-endian count of the number of failed boots.
Once it hits eight your Nook will boot into the recovery image uRecImg, uRecRam.
You could also echo about anything to that file to make it arithmetically greater than 8.
Code:
echo 000 > /rom/devconf/BootCnt
That is 0x0a303030 > 8
Normally this is a B&N thing that asks you about factory restore.
If you replaced those two files it could be Clockwork Mod Recovery.

Backup your Data! One seems to be zero'ing deleted content right away

I accidentally deleted a folder on my One's internal storage ( "/storage/sdcard0" which is also known as "/sdcard/" and linked to "/storage/emulated/0/" ) yesterday. There have been some documents that were deleted so I tried to recover them.
Here on xda we have a comprehensive guide how to recover internal storage. The problem is that other than SD card based external storage, internal storage is ext4 formatted and not visible to a computer as mass storage (content is transferred via MTP) hence common recovery tools such as testdisk, recuva, r-studio can't search the file system on a low level to recover lost files from an android's internal storage.
The solution to this is to copy over the entire (raw) data partition to your computer where you can proceed to have recovery tools search for your lost data. This can be done via adb. BusyBox and Root is required.
The detailed recovery/imaging process is described here:
http://forum.xda-developers.com/showthread.php?t=2143188
http://forum.xda-developers.com/showthread.php?t=1994705
BusyBox includes the linux "dd" command to create a bit-to-bit image of your data partition and "nc" that is used to pipe the data directly off the phone to your computer so nothing will get overwritten.
The userdata partition on the One is /dev/block/mmcblk0p37 and roughly about 25GB on 32GB Ones or 55GB on a 64GB One in size. It takes some hours to copy it over to your computer (was around 5h for my 64gig variant, 4MB/s avg).
After finishing that, you let multiple recovery tools of your choice work their magic on the huge raw partition backup, just to realize that other than existing data no deleted files are found.
Time to investigate further so I took another One which had the 4.2.2 based ARHD 10.2 installed, factory reset the device and filled it up with data. Then I deleted the whole userdata via my MTP-connected computer and immediately started over the imaging process. A few hours later I hex-edited the partition backup and found that 80% of the partition was zeros. My recovery tools still found existing data in /data/ that obviously have not been touched by me when deleting the "internal storage" through my computer as it is not accessible that way.
From my investigation so far, to me it seems that there is some kind of TRIM'ming or instant garbage collection going on to clean up unused NAND cells which carry previously deleted data. That is what is done on SSDs to maintain write speeds, which is a good thing actually. If that is the case, there would be virtually no chance to recover lost data as it gets wiped/zero'ed immediately.
What I want to find out is whether this behavior is from kernel or hardware, specifically internal flash controller. What are your thoughts?
Hope that helps others a bit.

Most phone NANDS have Trim activated. The HoX however did not, and that's why apps like FSTrim exists, to force a TRIM pass. Greatly smoothed out the random I/O lags even when scrolling through Sense.

If your using elementalx kernel it runs an fstrim command on boot and installs the binary when you flash the kernel ....
Its from an fstrim init.d mod I made for the hox which is being used in that kernel on the m7...
As for fstrim if your not using that kernel, I'm sure unless the fstrim file is in system/bin / xbin htc/Qualcomm arnt using it.

backfromthestorm said:
If your using elementalx kernel it runs an fstrim command on boot and installs the binary when you flash the kernel ....
Its from an fstrim init.d mod I made for the hox which is being used in that kernel on the m7...
As for fstrim if your not using that kernel, I'm sure unless the fstrim file is in system/bin / xbin htc/Qualcomm arnt using it.
Click to expand...
Click to collapse
I checked it and the file is not there. I didn't use elementalx kernel or any non-stock kernel when it happened. I did not reboot the phone before I made the image with dd.
There must be something in the stock kernel or nand controller...

Are the partitions mounted with the discard option?

flar2 said:
Are the partitions mounted with the discard option?
Click to expand...
Click to collapse
Indeed the data partition is mounted with discard. That might explain it. The file system reports free space to the controller immediately which kicks off the cleaning.