First things first:
1. This is not a thread about how to flash ROMs, recoveries or use Android SDK and ABD or fastboot commands. If you have questions about the basics, please keep them in @Funk Wizard's excellent thread created for that purpose:
OnePlus 5T: Unlock Bootloader | Flash TWRP | Root | Nandroid & EFS Backup & More !!
2. This thread is not the place to discuss the merits of encryption or decryption.
3. I'm not responsible for what you do to your own device. Read, think, read more, re-think, wipe, flash in that order.
4. This OP and the following posts will be updated as the discussion develops, so please check back here from time to time.
Now on to the discussion
There has been a lot of talk lately about encryption, decryption and the benefits and liabilities of each. Obviously having your device encrypted is a gain for security, something we should try to keep if possible. But encryption methods can vary, which is a problem for flashaholics like myself. When you flash a new ROM that cannot read the encryption of the previous ROM, /data must be formatted, causing the loss of /sdcard - pictures, music, files, etc.
Understanding the Problem
The issue seems to revolve around Qualcomm's "KeyMaster" encryption keys. While both Nougat and Oreo use FBE (File Based Encryption), by default they use different encryption keys, as pointed out by dev @codeworkx -- Nougat and Oreo 8.0 use KeyMaster 1 while Oreo 8.1 uses KeyMaster 3. So when an Oreo 8.1 ROM is flashed, it either can't access /data (requires decryption or formatting /data) or the ROM reformats /data itself, like early beta Lineage 15.1 builds. Likewise, reverting to a Nougat or Oreo 8.0 build will cause the same problem. Apparently, moving to KeyMaster 1 to 3 works (ie, flashing from OOS to Omni/Lineage) but reverting from Keymaster 3 back to 1 doesn't. When this happens, OOS can still decrypt with your PIN/password but TWRP can't.
One solution is to run unencyrpted, for which you may find threads in the How-To section. This discussion is about how to stay encrypted and flash back & forth between ROMs without loosing all of your data.
Links on the subject:
https://source.android.com/security/encryption/file-based
I look forward to your contribution to this discussion! :good:
Reserved
Just dropping this here:
mad-murdock said:
If only someone would be advanced in linux FBE, used tools and libraries. There surely is a way to remove encryption with a flashable .zip. _IF_ current TWRP has the needed tools onboard.
I hope one day we get encrypt/decrypt options in TWRP - where it belongs.
Click to expand...
Click to collapse
Yes, NOW I have seen this thread. Thanks for mentioning.
Seems useful.
After a bit of google kicking, I found this: https://source.android.com/security/encryption/file-based
Seems a good start on the topic. Maybe add it to a list of (hopefully growing) links?
Wow. Seems like this didn't work out that well.
mad-murdock said:
Wow. Seems like this didn't work out that well.
Click to expand...
Click to collapse
1. Rather than understand and deal with it, lots of people decrypt.
2. The issue hasn't gone away. Give it time.
Great information to those who recently owned an OP even if they have knowledge how to flash ROMs. (Including me)
Thanks!
I've stumbled across another issue for investigation. While experimenting yesterday, I discovered that @codeworkx TWRP 3.2.1-0 for Oreo (8.0 and 8.1) is able to read stock OOS/OOS B1 encryption until it is backed up in TWRP, an Oreo 8.1 ROM is flashed (eg, Omni, Lineage), and OOS is restored. After that, TWRP cannot decrypt /data with the correct PIN/password of the restored OOS ROM or "default_password". It doesn't matter if the nandroid was taken with or without a PIN/password, if the PIN/password is removed from the Oreo 8.1 ROM before restoring the nandroid, etc. Codeworkx suspects it has to do with how the passwords are being stored between 8.0 and 8.1.
And before you ask, I never formatted /data or decrypted & re-encrypted. The contents of /sdcard survived every flash and nandroid restore. Every ROM flashed was able to access /data with the correct PIN/password including the restored OOS. Only TWRP can't read /data with the correct password.
Also, I've left recovery systemless. That means my nandroid backups are only of data, and I restore by flashing the stock OOS ROM and only restoring the data nandroid. So zero changes have been made to system.
the Doctor said:
I've stumbled across another issue for investigation. While experimenting yesterday, I discovered that @codeworkx TWRP 3.2.1-0 for Oreo (8.0 and 8.1) is able to read stock OOS/OOS B1 encryption until it is backed up in TWRP, an Oreo 8.1 ROM is flashed (eg, Omni, Lineage), and OOS is restored. After that, TWRP cannot decrypt /data with the correct PIN/password of the restored OOS ROM or "default_password". It doesn't matter if the nandroid was taken with or without a PIN/password, if the PIN/password is removed from the Oreo 8.1 ROM before restoring the nandroid, etc. Codeworkx suspects it has to do with how the passwords are being stored between 8.0 and 8.1.
And before you ask, I never formatted /data or decrypted & re-encrypted. The contents of /sdcard survived every flash and nandroid restore. Every ROM flashed was able to access /data with the correct PIN/password including the restored OOS. Only TWRP can't read /data with the correct password.
Also, I've left recovery systemless. That means my nandroid backups are only of data, and I restore by flashing the stock OOS ROM and only restoring the data nandroid. So zero changes have been made to system.
Click to expand...
Click to collapse
""And before you ask, I never formatted /data or decrypted & re-encrypted. The contents of /sdcard survived every flash and nandroid restore. Every ROM flashed was able to access /data with the correct PIN/password including the restored OOS. Only TWRP can't read /data with the correct password""
So Do you mean to say I can revert back to OOS OB-1 by flashing it over Omni/LOS/etc via TWRP without formatting Data, and later on restoring Nandroid data of OOS OB-1.
shail139 said:
""And before you ask, I never formatted /data or decrypted & re-encrypted. The contents of /sdcard survived every flash and nandroid restore. Every ROM flashed was able to access /data with the correct PIN/password including the restored OOS. Only TWRP can't read /data with the correct password""
So Do you mean to say I can revert back to OOS OB-1 by flashing it over Omni/LOS/etc via TWRP without formatting Data, and later on restoring Nandroid data of OOS OB-1.
Click to expand...
Click to collapse
Yes, but obviously TWRP would not be able to decrypt with a PIN/password set by OOS. That just means you would have to disable lockscreen protection in the ROM before going into TWRP.
the Doctor said:
Yes, but obviously TWRP would not be able to decrypt with a PIN/password set by OOS. That just means you would have to disable lockscreen protection in the ROM before going into TWRP.
Click to expand...
Click to collapse
By that way the steps to restore should be...
1. Backup of OOS OB-1 in TWRP should be taken post removal all securities PIN/PASSWORD/etc (On external drive/OTG)
2. Flash OOS OB-1 normally, clean flash, boot to system, no security should be set
3. Boot to TWRP, restore OOS OB-1 Backup Only "Data" should be checked via OTG drive
4. Reboot to system
"twrp-3.2.1-0-universal-codeworkx-dumpling" will be the TWRP to be used
Correct me if I am wrong in steps
so in this case, am i right to say that, so long i dont do nandroid restore, i wouldnt have problem with encryption/decryption regardless of what rom i'm flashing using codeworkx's universal TWRP?
usually i always clean flash new roms and i'm ok to go through the 'hassle' of reinstalling stuffs. if i want to go back to the previous rom, i'll just do a clean flash of the previous rom instead of reverting back via nandroid.
so technically so long i'm on the right TWRP, i'm fine with switching roms am i right?
thanks for sharing the findings as well!
gorillaCF said:
so in this case, am i right to say that, so long i dont do nandroid restore, i wouldnt have problem with encryption/decryption regardless of what rom i'm flashing using codeworkx's universal TWRP?
usually i always clean flash new roms and i'm ok to go through the 'hassle' of reinstalling stuffs. if i want to go back to the previous rom, i'll just do a clean flash of the previous rom instead of reverting back via nandroid.
so technically so long i'm on the right TWRP, i'm fine with switching roms am i right?
thanks for sharing the findings as well!
Click to expand...
Click to collapse
I tried a clean flash of OOS from TWRP as well, but even that didn't work. I think you'd have to restore factory encryption per this guide to get TWRP to be able to decrypt OOS again:
[How To] Revert to 100% stock OOS from Oreo 8.1 | Restore factory encryption
Again, you can flash, backup and restore in TWRP even if you don't. It just won't be able to decrypt /data with your OOS PIN/password, so you'd have to remove lockscreen security first.
the Doctor said:
I tried a clean flash of OOS from TWRP as well, but even that didn't work. I think you'd have to restore factory encryption per this guide to get TWRP to be able to decrypt OOS again:
[How To] Revert to 100% stock OOS from Oreo 8.1 | Restore factory encryption
Again, you can flash, backup and restore in TWRP even if you don't. It just won't be able to decrypt /data with your OOS PIN/password, so you'd have to remove lockscreen security first.
Click to expand...
Click to collapse
Formating /data is the only way to go back to 8.0 crypto (after booting fully stock) and then you can use you Nandroids from OOS to restore /data with PIN, face unlock all ON.
Been there, done that from 8.1 custom to OOS N.
Didn't use stock recovery, didn't use revert builds, there actually were none at the time, but I think they are unneeded anyway.
It's a cumbersome process because backing up internal storage and restoring it is a pain when you have a lot of data to carry around.
But it's pretty straight forward.
All this done on blu_spark TWRP.
The problem I noted above wasn't that OOS couldn't read or encrypt /data properly after the nandroid backup--TWRP couldn't read OOS's PIN/password. I had no problems restoring and running OOS after running Omni/Lineage. After I restored OOS, on first boot I entered the PIN and found that my fingerprints and face unlock still worked. But when I booted back into Codeworkx TWRP neither the PIN or "default_password" worked. I didn't try Blu_Spark.
IMO, what we ultimately want is an official TWRP that can decrypt without workarounds so we can avoid the cumbersome process or formatting /data and moving everything back to /sdcard.
Edit: Here is the exact sequence of what happened:
I came from OOS OB1 with /data formatted by the stock recovery, encrypted, with PIN/fingerprints/face unlock.
I booted Codeworkx recovery, entered the PIN, it decrypted properly, I did a nandroid backup of the Data partition.
Still in recovery, I wiped Dalvik-Art/Cache/System/Data, then flashed Omni, gapps, Magisk.
I ran Omni for a while, moved to Lineage using the same process as above. I never removed the PIN, and Codeworkx TWRP had no problems decrypting with it in Omni or Lineage.
After running Lineage for a while, I went back into Codeworkx TWRP, decrypted with my PIN (it worked), wiped as above, flashed OOS OB1 with the factory zip, wiped the Data partition, restored Data from nandroid, flashed Magisk, rebooted.
On first boot OOS asked for a PIN. I entered my PIN and found my fingerprints & face unlock still working.
VVV HERE IS THE PROBLEM STARTED VVV
When I booted back into Codeworkx TWRP it could not decrypt with my PIN. I booted back into OOS and removed my PIN, set lockscreen protection to "None". TWRP still could not decrypt /data. I tried "default_password" but no dice.
Revert back to Omni, remove PIN, reboot TWRP, still can't decrypt.
So something changed between when I restored OOS OB1 (TWRP could decrypt with the PIN) and after first boot (TWRP couldn't decrypt with the PIN). Also, why could TWRP decrypt with OOS OB1's PIN to do the nandroid backup from a clean flash and to restore the same backup after being on Omni/Lineage, but couldn't decrypt with it after the first boot of the OOS nandroid backup?
Again, formatting /data again is not an acceptable workaround. I think we want to understand what changed and solve the problem.
the Doctor said:
Again, formatting /data again is not an acceptable workaround. I think we want to understand what changed and solve the problem.
Click to expand...
Click to collapse
The mentioned /data format is not a workaround per se, it's the only working workflow to get things going once you find the need to get back to OOS for the time being.
Accepting that is part of the process!
Users should know this upfront so they don't find out the hard way.
I'm currently running OxygenOS 5.0.3 and my understanding is that it uses Keymaster1. If I'm now upgrading to LineageOS 15.1 it'd change to Keymaster3 but without the need of formatting.
However, if I'd want to revert to OxygenOS 5.0.3 with Keymaster1 I would have to format /data. Is my understanding correct?
Macusercom said:
I'm currently running OxygenOS 5.0.3 and my understanding is that it uses Keymaster1. If I'm now upgrading to LineageOS 15.1 it'd change to Keymaster3 but without the need of formatting.
However, if I'd want to revert to OxygenOS 5.0.3 with Keymaster1 I would have to format /data. Is my understanding correct?
Click to expand...
Click to collapse
My experience has been that the ROM can decrypt without any issues, but TWRP can't decrypt without formatting /data with the stock recovery.
the Doctor said:
My experience has been that the ROM can decrypt without any issues, but TWRP can't decrypt without formatting /data with the stock recovery.
Click to expand...
Click to collapse
As previous posters have alluded to, use "twrp-3.2.1-0-universal-codeworkx-dumpling.img". This is able to decrypt 5.0.3.
wunderdrug said:
As previous posters have alluded to, use "twrp-3.2.1-0-universal-codeworkx-dumpling.img". This is able to decrypt 5.0.3.
Click to expand...
Click to collapse
Right. Flash Omni or Lineage, then go back OOS and try it again as Macusercom says in the post I quoted.
Here is a quick review of what happened:
The phone was running well on Lineageos 14.1, encrypted with a pattern, with TWRP 3.1.1 as recovery. I’ve then been notified of an LOS update to 15.1 Oreo and got excited. Booted in recovery, entered my pattern to decrypt, took a full TWRP backup, flashed Oreo modem and firmware and dirty flashed (I know, bad idea) the LOS 15.1 zip.
It got stuck at boot logo. Even if I didn't have high expectations for it to work I just thought I would try the lazy way to see and use my fresh backup to restore in case of failure like I always did successfully since my Nexus One.
This time was different because as I rebooted in TWRP to restore, it didn't ask for my pattern. You guessed it, the data partition is encrypted, no access to my backup or anything on the external storage. I can mount data and see the weird encrypted file names but that's it. I tried different version of TWRP, but it never ask the pattern. Even the terminal command 'twrp decrypt *******' doesn’t work.
I then tried to wipe data and flash LOS 14.1 again but it gets stuck saying that android has no access to data partition because it’s encrypted and that I need to format. I pulled out my sim card and started to use my old oneplus one while waiting for the new version of TWRP 3.2.1.1 with the feb security patch support thinking it might then be able to decrypt but no luck still.
I can go without the phone for a while, I’ll buy another one if I have to because there is some precious data on that phone and I can’t make my mind that the data is there, I know the encryption key but I have no access to it. There must be a way, I just don’t have enough knowledge about how this encryption thing is working.
Any help would be appreciated, Thank you
jpitou said:
Here is a quick review of what happened:
The phone was running well on Lineageos 14.1, encrypted with a pattern, with TWRP 3.1.1 as recovery. I’ve then been notified of an LOS update to 15.1 Oreo and got excited. Booted in recovery, entered my pattern to decrypt, took a full TWRP backup, flashed Oreo modem and firmware and dirty flashed (I know, bad idea) the LOS 15.1 zip.
It got stuck at boot logo. Even if I didn't have high expectations for it to work I just thought I would try the lazy way to see and use my fresh backup to restore in case of failure like I always did successfully since my Nexus One.
This time was different because as I rebooted in TWRP to restore, it didn't ask for my pattern. You guessed it, the data partition is encrypted, no access to my backup or anything on the external storage. I can mount data and see the weird encrypted file names but that's it. I tried different version of TWRP, but it never ask the pattern. Even the terminal command 'twrp decrypt *******' doesn’t work.
I then tried to wipe data and flash LOS 14.1 again but it gets stuck saying that android has no access to data partition because it’s encrypted and that I need to format. I pulled out my sim card and started to use my old oneplus one while waiting for the new version of TWRP 3.2.1.1 with the feb security patch support thinking it might then be able to decrypt but no luck still.
I can go without the phone for a while, I’ll buy another one if I have to because there is some precious data on that phone and I can’t make my mind that the data is there, I know the encryption key but I have no access to it. There must be a way, I just don’t have enough knowledge about how this encryption thing is working.
Any help would be appreciated, Thank you
Click to expand...
Click to collapse
Try flash codeworkx TWRP ...
it should decrypt your data partition ...
https://downloads.sourceforge.net/project/cheeseburgerdumplings/15.1/cheeseburger/recovery/twrp-3.2.1-0-20180309-codeworkx-cheeseburger.img?r=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fcheeseburgerdumplings%2Ffiles%2F15.1%2Fcheeseburger%2Frecovery%2F&ts=1521806282&use_mirror=netix
PS-DEV said:
Try flash codeworkx TWRP ...
it should decrypt your data partition ...
https://downloads.sourceforge.net/project/cheeseburgerdumplings/15.1/cheeseburger/recovery/twrp-3.2.1-0-20180309-codeworkx-cheeseburger.img?r=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fcheeseburgerdumplings%2Ffiles%2F15.1%2Fcheeseburger%2Frecovery%2F&ts=1521806282&use_mirror=netix
Click to expand...
Click to collapse
I had tried it when it came out and I just tried it again. No difference!! It looks like twrp doesn't even see that my phone is encrypted. I've read a lot and tried many different things and I'm out of idea. All of the people I've seen with this problem have given up so they could get their phone back and running by formatting the partition, losing their data. This option is not one for me. I'd rather buy a new phone hoping for an eventual possible solution. I know the data is there, and I know the pattern key...... I mean, there's got to be a way......
Hi, ive been wondering why all those roms folks put out are unable to encrypt the /data partition.
Any rom i tried so far is soft rebooting after i invoked the encryption option. No encyption happening at all.
And if i do it manually via adb shell > su # vdc cryptfs enablecrypto wipe password somethingpw
I sort of soft brick the device boot until i whipe the data partion via twrp.
Thereof i am wondering why its broken in the first place and if theres a chance if folk will fix this over time or any 3rd Party Roms will be stuck w/o a chance to encrypt personal data.
Myau said:
Hi, ive been wondering why all those roms folks put out are unable to encrypt the /data partition.
Any rom i tried so far is soft rebooting after i invoked the encryption option. No encyption happening at all.
And if i do it manually via adb shell > su # vdc cryptfs enablecrypto wipe password somethingpw
I sort of soft brick the device boot until i whipe the data partion via twrp.
Thereof i am wondering why its broken in the first place and if theres a chance if folk will fix this over time or any 3rd Party Roms will be stuck w/o a chance to encrypt personal data.
Click to expand...
Click to collapse
I was facing issues with my LOS 15.1, afaik unencrypted. Then I flash stock Oreo Feb using Mi Flash tool, with option 'clean all and lock'. Then I root it with magisk and flash LOS 15.1. The next thing I know, my phone is encrypted
kopitalk said:
I was facing issues with my LOS 15.1, afaik unencrypted. Then I flash stock Oreo Feb using Mi Flash tool, with option 'clean all and lock'. Then I root it with magisk and flash LOS 15.1. The next thing I know, my phone is encrypted
Click to expand...
Click to collapse
Did you dirty flash or whipe the installed OS?
Myau said:
Did you dirty flash or whipe the installed OS?
Click to expand...
Click to collapse
Clean install, i.e. wipe all.
kopitalk said:
Clean install, i.e. wipe all.
Click to expand...
Click to collapse
Thank You, for confirming that Encryption works, i was almost at the verge of returning the phone.
Got it to work now.
....
The tldr of all i tested and messed around with is:
If you whipe Data trough twrp, with the special dialog where you have to type in YES. which folk reccomend to use to remove encryption,
will mean you brick the encryption and requires a re-flashing of the stock rom.
Secondly if you only go into advanced whipe, Art, System and data and then install a rom, it should say 'decrypted with default password'
Which is a dead give away that encryption might going to work just fine, rebooted into bootloader w/o into system, installed gapps pico, (eeh) and magisk, which also mentioned something about encryption as the log ran trough. Then rebooted into system, and now it says encrypted. And i can set up a boot pin/password before it fully boots. GREAT!
Soo.... If it says decrypted and won't encrypt, you have to flash stock, check its encrypted. Then install costum rom without removing encryption.
Once you removed it, its back to stock rom for new a /data setup/encryption.
So, after a few months on Oreo I clean flashed DotOS Pie last night, got the whole thing set up and tried to encrypt phone. It now bootloops, but I can get to TWRP, which shows me a pattern to decrypt data, even though my phone had PIN lock. I did the trick to "translate" between PIN and pattern grid. TWRP shows me "data successfully decrypted, new block device: '/dev/block/dm-0'", but I can't mount /data (consequently can't mount /sdcard), though I can mount /system. TWRP's console shows "Failed to mount /data (Invalid argument)".
I know I can just format /data on the wipe menu to get rid of encryption, but since I don't have a backup of /sdcard, I would like to know if there is any way to get /data mounted again so I can backup my internal storage before formatting /data.
If it makes any difference, my TWRP version is 3.2.1-0, latest one available at twrp.me. I heard that 3.2.1-1 handles encryption better (even better than more recent versions on some devices), but it's not available on the official site.
--Other details that might be relevant, but I'm not sure:
1. I do have TiBu's backup on my computer, so if I must restore my apps I'm fine with it (though I don't have a backup of the whole internal storage);
2. I also have a Nandroid backup of /system, /data and /boot, but it's on the phone storage only;
3. Last, but not least, I have flashed latest Magisk just before encrypting the device.
I saw some stuff related to a no-verity encryption zip file to be flashed via TWRP in some discussions regarding other devices, would it be of any help in my case?
@anupritaisno1 sorry to bother you, but you're the one person to ask for help with TWRP on OP2. Could you lend me a hand here? Would your enhanced TWRP help in my case? I know one shouldn't restore backups made with one version of TWRP within another version, and also that the tempcache stuff needs to be set up to flash ROMs. My plan here, if this could help, was to see if your enhanced TWRP can successfully decrypt my /data and internal storage, then I would back up my internal storage to my computer and either set up tempcache or reflash official 3.2.1-0 to format data and flash the ROM again.
Thanks in advance.