Related
There are a lot of toturials for building CM9 on ubuntu or CM7 on Mac but I couldn't find a decent toturial for building CM9 on Mac (specially Lion). Development in AOSP/CM land is rapid and guides frequently need updating. I had to spend a little time to figure everything out and I decided to share it here.
This tutorial is for building CM9 (ICS) for Galaxy Nexus GSM (maguro) on Mac OS X Lion 10.7.3 using Xcode 4.3 and homebrew . You can easily make the instructions work for most other cm9 devices, but I wouldn't know anything about that.
DISCLAIMER: I'm not responsible if you blow yourself up, blah blah blah
However, I've tried to make this as noob friendly as possible because, well I'm a noob myself
Instrunctions:
UPADTE (MAY 29TH) : The Xcode 4.3 default compiler (llvm-gcc) used to be incompatible with CM9. Thanks to jocelyn and topprospect, the LLVM compatibility patches from mainline AOSP are now merged into CM9. Therefore, you can now use Xcode 4.3 and its command line tools without installing another compiler. However, since GCC is still the only officially supported compiler, incompatibilites with llvm-gcc could still be introduced with future updates. Therefore, if your build fails, it might be worth it to try installing and compiling with GCC 4.2. See the Troubleshooting section for more info.
Now that we have Xcode 4.3 and Xcode command line tools (CLT) installed, let's continue.
Open Terminal and run
Code:
java
if you don't have Java, you will get a prompt asking you to download and install Java. Go ahead and install it.
If you don't have adb and fastboot working, download the android-sdk from google (version r18 as of now) and put it in /usr/local/ and rename the folder to "android-sdk".
Install the homebrew package manager
Code:
/usr/bin/ruby -e "$(/usr/bin/curl -fsSL https://raw.github.com/mxcl/homebrew/master/Library/Contributions/install_homebrew.rb)"
To make sure that homebrew and android-sdk executables are in $PATH:
Code:
touch ~/.bash_profile && echo "PATH=/usr/local/bin:/usr/local/sbin:$PATH:/usr/local/android-sdk/tools:/usr/local/android-sdk/platform-tools" >> ~/.bash_profile
Relaunch Terminal for the change to take effect.
At this point you can run
Code:
brew doctor
to detect any problems there might be (Homebrew may instruct you to use the xcode-select utility to select the xcode installation path). Hopefully, your system is raring to brew
Now we have to install a bunch of packages:
Code:
brew install git coreutils findutils gnu-sed gnupg pngcrush repo
We now need to create a couple of symlinks so that the gnu versions of 'sed' and 'find' are used rather than the osx provided versions :
Code:
ln -s /usr/local/bin/gfind /usr/local/bin/find && ln -s /usr/local/bin/gsed /usr/local/bin/sed && ln -s /usr/local/bin/gstat /usr/local/bin/stat
It's time to create a case sensitive image which will hold our working directory:
Code:
hdiutil create -type SPARSE -fs "Case-sensitive Journaled HFS+" -size 40g -volname "android" -attach ~/Desktop/Android
Now we have a disk image in ~/Desktop/Android. Mount it if it's not mounted already. (Don't be picky about the size, the image will only take as much as space as its contents).
Now we need to create a working directory inside the mounted volume:
Code:
cd /Volumes/android && mkdir cm9 && cd cm9
We can initialize and download the source now:
Code:
repo init -u git://github.com/CyanogenMod/android.git -b ics && repo sync && say 'finished'
Now we need to get the required proprietry files for our device. We can get these from the device itself. Connect your phone (make sure USB Debugging is enabled) and run the following (for maguro):
Code:
cd /Volumes/android/cm9/device/samsung/maguro/ && ./extract-files.sh
[If you see errors in the output from extract-files.sh, see the Troubleshooting section below]
For Google devices,we can also get them directly from google. For maguro, download the 3 files and extract them to /Volumes/android/cm9. Then,
Code:
cd /Volumes/android/cm9
/Volumes/android/cm9/extract-broadcom-maguro.sh
/Volumes/android/cm9/extract-imgtec-maguro.sh
/Volumes/android/cm9/extract-samsung-maguro.sh
We also need the prebuilts (like ROM manager and Term.apk):
Code:
/Volumes/android/cm9/vendor/cm/get-prebuilts
You can optionally tell the build to use the ccache tool. CCache acts as a compiler cache that can be used to speed-up rebuilds :
Code:
export USE_CCACHE=1 && /Volumes/android/cm9/prebuilt/darwin-x86/ccache/ccache -M 20G
Default is 1GB. Anything between 20GB-50GB should be fine.
Before starting the build, we need to workaround an issue with Lion and compiling the QEMU emulator.
[This step doesn't seem to be needed anymore. QEMU is automatically ignored on OS X/Darwin]
If you build now, you're probably gonna get kernel build errors regarding the missing elf.h header (this error might be device specific). Fortunately, we already have this file downloaded, so we only need to copy it to /usr/local/include:
Code:
cp /Volumes/android/cm9/external/elfutils/libelf/elf.h /usr/local/include
FINALLY, we are ready to build:
Code:
cd /Volumes/android/cm9 && source build/envsetup.sh && brunch
Pick your device from the list and enter the number. For maguro, you could use "brunch maguro" instead and skip the menu. Depending on your system, this will take 30min-4hours.
You should now see a beautiful zip file waiting to be flashed:
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Troubleshooting:
The extract script for maguro seems to be a little outdated, as it doesn't pull the gps proprietary blob. You can either use the google provided scripts, or add koush's git repository for your device to your local_manifest.xml.
As explained above, the CM9 source is currently compatible with llvm-gcc. In the future, if llvm-gcc fails to build correctly, you should try installing and compiling using GCC4.2 (the Xcode 3 compiler). You can install apple-gcc4.2 from homebrew:
Code:
brew install https://raw.github.com/Homebrew/homebrew-dupes/master/apple-gcc42.rb
This version of gcc can happily coexist with Xcode 4.x .
So now you have GCC 4.2 installed, but it won't be used unless we update the corresposing environment variables:
Code:
export CC=/usr/local/bin/gcc-4.2 && export CXX=/usr/local/bin/g++-4.2
Notice that using the export command is temporary. If you relaunch Terminal, you will need to set these again. However, this is a good thing, because changing these values permanently (by putting them in ~/.bash_profile) can interfere with other builds.
If you use this method, the build might fail while compiling "external/zlib/x86/adler32.c". It appears that a recent change in zlib has introduced an incompatibility with gcc 4.2. you have to revert the following 2 commits:
Code:
cd external/zlib
git revert dd6786cae3f4493faa6661d5f74db587932f15d7
git revert 13bf40af68236c961542bdee1d4b7c0176bf15a0
Alternatively, you can add topprospect's zlib on github (which has those commits reverted) to your local_manifast.xml. Simply run:
Code:
nano /Volumes/android/cm9/.repo/local_manifest.xml
and add the following line
Code:
<project name="dferg/android_external_zlib" path="external/zlib" remote="github" />
If you get a build error, and your error is not covered here, copy the last 20-30 lines of the build output AND the output from the following command into pastebin and post the link. Hopefully me or someone else will help you.
Code:
echo -e "\nENV:\n$(env)\n\nWHICH GCC\n:$(which gcc)\n\nWHICH G++:\n$(which g++)\n\nWHICH CC:\n$(which cc)\n\nWHICH C++:\n$(which c++)\n\nBREW DOCTOR:\n$(brew doctor)\n\nBREW LIST:\n$(brew list)\n\n/USR/BIN:\n$(ls -l /usr/bin | grep gcc)\n\n/USR/LOCAL/BIN:\n$(ls -l /usr/local/bin | grep gcc)\n\n"
Notes/Extras:
To quickly setup your environment, add an alias like the following to ~/.bash_profile:
Code:
alias cm9env="hdiutil attach PATH-TO-DISK-IMAGE -mountpoint /Volumes/android && cd /Volumes/android/cm9 && source ./build/envsetup.sh && export USE_CCACHE=1"
alias cm9build="cm9env && make clobber && reposync && brunch maguro"
Now you can save time by using "cm9env" to get your environment setup or "cm9build" to compile a clean updated build.
To clear your output directory for a new build, run "make clobber". You probably don't need this if you've only changed a few lines of code.
To cherry pick yet-to-be-merged changes from the gerrit instance:
1. Pick an open commit from CM Gerrit
2. Under list of Patch Sets pick the latest and open cherry-pick tab
3. Check what Git repository the url is pointing e.g. http://review.cyanog...frameworks_base
4. In your CM9 working tree go to the corresponding directory, which in this case is something like ~/your-working-directory/frameworks/base/
5. Now simply paste the whole line seen in CM Gerrit cherry-pick tab e.g. "git fetch http://review.cyanog....rameworks_base refs/changes/00/13100/4 && git cherry-pick FETCH_HEAD"
It should be now included in your next compiled build. When doing repo sync again, cherry picks will be lost.[CREDIT Fihlvein from xda]
Click to expand...
Click to collapse
Enjoy your custom built CM9!
I updated OP with some updated info about Xcode 4.3.
Nice work man. Very helpful.
conantroutman said:
Nice work man. Very helpful.
Click to expand...
Click to collapse
thanks! I'll try to keep this topic updated as issues with mac are introduced/resolved.
Nice write up. Thanks.
Before I start again from scratch I have a question. Does this guide apply to previous versions of mac os x (mine is 10.6.8)? I used the official android initializing build environment page & cm7 wiki page for the instructions to setup my build environment.
Also, any tips to switch from macports to homebrew?
In the past I've had to cherry pick to get my OS X build environment set up for CM9. The compile from source fails because I started with macports instead of homebrew (bad idea). I tried to switch to homebrew without success. Any tips to switch from macports to homebrew?
For the sake of keeping this page on topic a pm response is ok if that is what you prefer.
Hi thanks for this !! Helpful one question what do i change so i can do AOSP instead of cm9??
grad061980 said:
Nice write up. Thanks.
Before I start again from scratch I have a question. Does this guide apply to previous versions of mac os x (mine is 10.6.8)? I used the official android initializing build environment page & cm7 wiki page for the instructions to setup my build environment.
Also, any tips to switch from macports to homebrew?
In the past I've had to cherry pick to get my OS X build environment set up for CM9. The compile from source fails because I started with macports instead of homebrew (bad idea). I tried to switch to homebrew without success. Any tips to switch from macports to homebrew?
For the sake of keeping this page on topic a pm response is ok if that is what you prefer.
Click to expand...
Click to collapse
Yes, this guide should work fine on Snow Leopard. It mostly depends on your Xcode version. If you have Xcode 3, you can skip step 1 entirely, since you already have gcc4.2 as part of Xcode.
If you have access to Xcode 4.2 and above, you will need to install gcc4.2 separately, as explained in the guide.
Now regarding Macports, I strongly suggest that you completely uninstall Macports before installing homebrew. Instructions are here: http://guide.macports.org/chunked/installing.macports.uninstalling.html
WonkyYew said:
Hi thanks for this !! Helpful one question what do i change so i can do AOSP instead of cm9??
Click to expand...
Click to collapse
Full instructions are available on android.com : http://source.android.com/source/initializing.html
If you are using this guide, you need to change the repo initialization command to :
Code:
repo init -u https://android.googlesource.com/platform/manifest
and then do repo sync. You can setup ccache as usual. I don't think AOSP has the "brunch command", so you have to use launch and then make.
Run "lunch" and select an option from the menu. You can find more info about the options here: http://source.android.com/source/building.html. For maguro, you should use "full_maguro-userdebug".
To start the build, use
Code:
make -j$(sysctl -n hw.ncpu)
@ArmanUV. Sounds good & thanks for the input. I'll give the link to macports uninstall a go.
Im a real noob, whats the advantage of compiling from source?
Thanks for the help man !
Hi, thanks for your guide, setting up the repo was no problem at all!
But: I'm getting the following error when building.
Code:
external/zlib/x86/adler32.c: In function ‘adler32_MMX’:
external/zlib/x86/adler32.c:747: error: can't find a register in class ‘GENERAL_REGS’ while reloading ‘asm’
external/zlib/x86/adler32.c:747: error: ‘asm’ operand has impossible constraints
make: *** [out/host/darwin-x86/obj/STATIC_LIBRARIES/libz_intermediates/adler32.o] Error 1
make: *** Waiting for unfinished jobs....
Seems to be a problem with the compiler, but I'm on xcode 4.3 and I've installed gcc-4.2 and set the env vars. Any help?
ArmanUV,
Thanks so much for posting this guide. Very helpful!
Are you having any trouble with errors like this?
Code:
external/zlib/x86/adler32.c: In function ‘adler32_MMX’:
external/zlib/x86/adler32.c:747: error: can't find a register in class ‘GENERAL_REGS’ while reloading ‘asm’
external/zlib/x86/adler32.c:747: error: ‘asm’ operand has impossible constraints
Googling for this error implies that the fix is to use a version of GCC > 4.2. But there does not seem to be a GCC 4.4 in Homebrew.
Thanks again for the guide!
EDIT: Sorry for the double post with empyyy. Seems like there is someone else having my same issue!
topprospect said:
ArmanUV,
Thanks so much for posting this guide. Very helpful!
Are you having any trouble with errors like this?
Code:
external/zlib/x86/adler32.c: In function ‘adler32_MMX’:
external/zlib/x86/adler32.c:747: error: can't find a register in class ‘GENERAL_REGS’ while reloading ‘asm’
external/zlib/x86/adler32.c:747: error: ‘asm’ operand has impossible constraints
Googling for this error implies that the fix is to use a version of GCC > 4.2. But there does not seem to be a GCC 4.4 in Homebrew.
Thanks again for the guide!
EDIT: Sorry for the double post with empyyy. Seems like there is someone else having my same issue!
Click to expand...
Click to collapse
empyyy said:
Hi, thanks for your guide, setting up the repo was no problem at all!
But: I'm getting the following error when building.
Code:
external/zlib/x86/adler32.c: In function ‘adler32_MMX’:
external/zlib/x86/adler32.c:747: error: can't find a register in class ‘GENERAL_REGS’ while reloading ‘asm’
external/zlib/x86/adler32.c:747: error: ‘asm’ operand has impossible constraints
make: *** [out/host/darwin-x86/obj/STATIC_LIBRARIES/libz_intermediates/adler32.o] Error 1
make: *** Waiting for unfinished jobs....
Seems to be a problem with the compiler, but I'm on xcode 4.3 and I've installed gcc-4.2 and set the env vars. Any help?
Click to expand...
Click to collapse
You guys seem to have the same issue. What sort of Xcode configuration are you using? Can you post the output from "which gcc","which g++", "gcc -v", "g++ -v" and "cc -v"?
ArmanUV said:
You guys seem to have the same issue. What sort of Xcode configuration are you using? Can you post the output from "which gcc","which g++", "gcc -v", "g++ -v" and "cc -v"?
Click to expand...
Click to collapse
I am on Lion 10.7.4 with Xcode 4.3.2. Here is the output that you asked for:
Code:
# echo -n "which gcc: "; which gcc; echo -n "which g++: "; which g++; echo ""; echo "gcc -v:"; gcc -v; echo ""; echo "g++ -v:"; g++ -v; echo ""; echo "cc -v:"; cc -v
which gcc: /usr/bin/gcc
which g++: /usr/bin/g++
gcc -v:
Using built-in specs.
Target: i686-apple-darwin11
Configured with: /private/var/tmp/llvmgcc42/llvmgcc42-2336.9~22/src/configure --disable-checking --enable-werror --prefix=/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2 --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-prefix=llvm- --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin11 --enable-llvm=/private/var/tmp/llvmgcc42/llvmgcc42-2336.9~22/dst-llvmCore/Developer/usr/local --program-prefix=i686-apple-darwin11- --host=x86_64-apple-darwin11 --target=i686-apple-darwin11 --with-gxx-include-dir=/usr/include/c++/4.2.1
Thread model: posix
gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.9.00)
g++ -v:
Using built-in specs.
Target: i686-apple-darwin11
Configured with: /private/var/tmp/llvmgcc42/llvmgcc42-2336.9~22/src/configure --disable-checking --enable-werror --prefix=/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2 --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-prefix=llvm- --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin11 --enable-llvm=/private/var/tmp/llvmgcc42/llvmgcc42-2336.9~22/dst-llvmCore/Developer/usr/local --program-prefix=i686-apple-darwin11- --host=x86_64-apple-darwin11 --target=i686-apple-darwin11 --with-gxx-include-dir=/usr/include/c++/4.2.1
Thread model: posix
gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.9.00)
cc -v:
Apple clang version 3.1 (tags/Apple/clang-318.0.58) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin11.3.0
Thread model: posix
They are pointing to LLVM, but my CC and CXX variables point to:
Code:
env|grep 4.2
CXX=/usr/local/bin/g++-4.2
CC=/usr/local/bin/gcc-4.2
Do I need to add something to PATH? I'm sure I just missed something obvious in your instructions.. Thanks for helping!
topprospect said:
I am on Lion 10.7.4 with Xcode 4.3.2. Here is the output that you asked for:
Do I need to add something to PATH? I'm sure I just missed something obvious in your instructions.. Thanks for helping!
Click to expand...
Click to collapse
Everything checks out. I'm away from my main machine so I can't run a test build, but I suspect that a recent change is causing problems.
Now, regarding the compiler, Setting CC/CXX *should* take care of everything, but I am currently not 100% sure that somewhere in a makefile, these environment variables aren't being ignored. Since I wrote the guide, I noticed a lot of clang warnings in the build, which means that CC/CXX is not honored and /usr/bin/cc and /usr/bin/c++ is being used.
A more robust method of making sure gcc-4.2 is being used is creating symlinks to gcc-4.2 and g++-4.2 :
Code:
ln -s /usr/local/bin/gcc-4.2 /usr/local/bin/gcc
ln -s /usr/local/bin/gcc-4.2 /usr/local/bin/cc
ln -s /usr/local/bin/g++-4.2 /usr/local/bin/c++
ln -s /usr/local/bin/g++-4.2 /usr/local/bin/g++
Obviousely, a systemic method like this has its downsides but it may be the only choice without having to change CM code (especially since I lack the knowledge to do so )
[I recently found out that master aosp is no longer using CC/CXX to find the compiler (see ./build/core/combo/). Instead, it uses "gcc" and "g++" directly, which means that llvm-gcc will be used no matter what env variable you have. Fortunately, unlike cm9, master aosp is supposed to build fine with llvm-gcc (except for qemu, which doesn't matter for device images). ]
ArmanUV said:
Everything checks out. I'm away from my main machine so I can't run a test build, but I suspect that a recent change is causing problems.
Now, regarding the compiler, Setting CC/CXX *should* take care of everything, but I am currently not 100% sure that somewhere in a makefile, these environment variables aren't being ignored. Since I wrote the guide, I noticed a lot of clang warnings in the build, which means that CC/CXX is not honored and /usr/bin/cc and /usr/bin/c++ is being used.
A more robust method of making sure gcc-4.2 is being used is creating symlinks to gcc-4.2 and g++-4.2 :
Code:
ln -s /usr/local/bin/gcc-4.2 /usr/local/bin/gcc
ln -s /usr/local/bin/gcc-4.2 /usr/local/bin/cc
ln -s /usr/local/bin/g++-4.2 /usr/local/bin/c++
ln -s /usr/local/bin/g++-4.2 /usr/local/bin/g++
Click to expand...
Click to collapse
Okay, just tried this. I created a new dir (/Volumes/Android/bin) that simply houses those softlinks you recommended. Then I put /Volumes/Android/bin at the beginning of my PATH. That should fix it without breaking the rest of the system, e.g. homebrew.
The GENERAL_REGS problem still exists though. Pretty sure b/c gcc 4.2.1 doesn't understand this construct properly (need a newer version of gcc).
So I backed out the change that introduced this adler32.c.
https://github.com/CyanogenMod/android_external_zlib/commit/13bf40af68236c961542bdee1d4b7c0176bf15a0
The compile is getting farther now. I have to run to work so I'll post later if it succeeds.
The weird thing is: This change was made back in December. Why would it have worked for you?
topprospect said:
The compile is getting farther now. I have to run to work so I'll post later if it succeeds.
Click to expand...
Click to collapse
Build works (and boots!) with the following:
Code:
cd external/zlib
git revert dd6786cae3f4493faa6661d5f74db587932f15d7
git revert 13bf40af68236c961542bdee1d4b7c0176bf15a0
Note the 1st revert is just to avoid massive conflicts seen when reverting the 2nd one by itself. The 2nd revert is the one that really matters here.
So this isn't really a solution.. Seems like we need to move to a newer version of gcc or figure out a patch to adler32.c that makes it gcc 4.2 compatible.
topprospect said:
Build works (and boots!) with the following:
Code:
cd external/zlib
git revert dd6786cae3f4493faa6661d5f74db587932f15d7
git revert 13bf40af68236c961542bdee1d4b7c0176bf15a0
Click to expand...
Click to collapse
The first revert works fine, however the second one gives me the following error:
Code:
$ git revert 13bf40af68236c961542bdee1d4b7c0176bf15a0
error: could not revert 13bf40a... Implement vectorized adler32 and optimized slhash
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
error: Could not parse conflict hunks in zlib.h
empyyy said:
The first revert works fine, however the second one gives me the following error:
Code:
$ git revert 13bf40af68236c961542bdee1d4b7c0176bf15a0
error: could not revert 13bf40a... Implement vectorized adler32 and optimized slhash
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
error: Could not parse conflict hunks in zlib.h
Click to expand...
Click to collapse
Well, run git status. You'll see that the only conflict is in the comments in zlib.h. So, you can just ignore it.
topprospect said:
Okay, just tried this. I created a new dir (/Volumes/Android/bin) that simply houses those softlinks you recommended. Then I put /Volumes/Android/bin at the beginning of my PATH. That should fix it without breaking the rest of the system, e.g. homebrew.
The GENERAL_REGS problem still exists though. Pretty sure b/c gcc 4.2.1 doesn't understand this construct properly (need a newer version of gcc).
So I backed out the change that introduced this adler32.c.
https://github.com/CyanogenMod/android_external_zlib/commit/13bf40af68236c961542bdee1d4b7c0176bf15a0
The compile is getting farther now. I have to run to work so I'll post later if it succeeds.
The weird thing is: This change was made back in December. Why would it have worked for you?
Click to expand...
Click to collapse
topprospect said:
Build works (and boots!) with the following:
Code:
cd external/zlib
git revert dd6786cae3f4493faa6661d5f74db587932f15d7
git revert 13bf40af68236c961542bdee1d4b7c0176bf15a0
Note the 1st revert is just to avoid massive conflicts seen when reverting the 2nd one by itself. The 2nd revert is the one that really matters here.
So this isn't really a solution.. Seems like we need to move to a newer version of gcc or figure out a patch to adler32.c that makes it gcc 4.2 compatible.
Click to expand...
Click to collapse
Nice find. I tried to compile this morning and I ran into the same issue. This is what I don't understand: I did a couple of builds about a week ago without running into this issue. But, the latest commits on zlib are from 2 months ago.
Amazingly, the Xcode 4.3 toolchain (clang and llvm-gcc) builds this external/zlib/adler32.c just fine.
An alternative to this problem is to install an up to date gcc 4.7 :
Code:
brew install https://raw.github.com/Homebrew/homebrew-dupes/master/gcc.rb
and then create symlinks to gcc-4.7/g++-4.7. I have not tested this yet.
THIS HELP REQUEST WAS ORIGINALLY POSTED IN THE DEV FORUM, BUT IT WAS "SUGGESTED" THAT IT BELONGS HERE INSTEAD.
Hi,
I'm trying to build CM7 Kernel from source. I've followed "Building_Kernel_from_source" in CM Wiki exactly. Executed yes "" | make oldconfig. I still get fatal script & compile errors. First was "usr/galaxysmtd_initramfs.list" not found. Created an empty one to get past this temporarily. Next got FIQ_START undefined. Tried manually fixing this (and others errors) to no avail. I think I'm missing some basic point. Can anyone offer any advice? Am I missing some patch? Is this a .config issue? Something else?
Thanks
Background (What I did):
dev machine: Ubuntu 10.04LTS
installed CM7.2 on phone.
installed development support tools and checked versions.
set up repo, downloaded from git://github/CyanomodGen/android.git -b gingerbread and repo sync.
set CCOMPILER.
checked that ARM EABI installed (Ok).
cloned kernel from git://github.com/CyanogenMod/cm-kernel.git.
adb pull config.gz from phone.
gzip -d -c > .config
Executed yes "" | make oldconfig (looked Ok)
Tried to build: make ARCH=arm CROSS_COMPILER=$CCOMPILER -j4
Got fatal script and compile errors (arrrgh).
This is a guide about getting a Linux distribution up and running on your Nexus 9. The goal of this thread is to give a basic understanding about how to get a basic system running, but not working throughout all the kinks that come with doing so on tablet style hardware.
This guide will be non-destructive, and won't touch the data on your tablet. Unless of course you've yet to unlock your bootloader, which if that is the case then shame on you.
Some prerequisites before starting this project with your Nexus 9
A Linux distribution running on your computer. Either in a virtual machine or as your main OS.
A USB flash drive
USB OTG cable
Some things that are nice to have
USB hub
USB keyboard
USB mouse
USB network card
Headphone UART cable
A brief overview about what we are going to be doing here.
First we are going to be building an initramfs. This is a small filesystem with a bare minimum set of tools that can be loaded in to RAM space and use. We will be using this while we are initially getting our kernel up and running, to make sure we didn't mess anything up.
Next we will move on to building the actual kernel for the device. We will be running a Android kernel since Nvidia hasn't been the quickest on their attempt to upstreaming everything required to run this device on a upstream Linux kernel. So we'll be running a 3.10 kernel, which is a heck of a lot better than some 3.0 or 3.4 thing that older devices are on.
Third we will be building our rootfs that will run from an external USB storage device, so we don't affect any data actually on the tablet. I may also add to the guide how to partition your internal storage space so we can use that instead.
[Step 0/4]
We need to set up our build environment on our host Linux distribution before we begin anything. I'm going to assume that the host Linux system we are running is Debian/Ubuntu based, so we'll be using apt to grab our packages we need.
Code:
sudo apt-get install gcc g++ git gcc-4.9-aarch64-linux-gnu g++-4.9-aarch64-linux-gnu libncurses5-dev
This installs the host gcc and AArch64 cross compiler, along with git. We will need these tools to build everything required.
Things that don't work.
Rebooting
Touchscreen
Way more things that people care about
The first step that we will be doing is building our filesystem that we will load in to memory for testing.
This is typically called an initramfs, which tends to be a compressed filesystem that gets copied in to a kernel image once the kernel is built. We won't be using this image for long, as it is mostly to make sure we have everything working correctly before jumping to a full linux distribution.
This step is technically unnecessary, but it brings you up to speed with what initially needs to be done to get something running.
So starting off, we are going to need to pick a target for the filesystem. I've decided to go with buildroot since it is a project that is really easy to get built.
Steps Overview
Clone buildroot with git
Setup buildroot to cross compile
Set up basic options
Compile!
1)*Clone buildroot from github.
Code:
git clone https://github.com/buildroot/buildroot.git
cd buildroot
2)*Start configuring buildroot
Code:
make menuconfig
This will bring up a menu for determining how we want to build our buildroot. We have quite a few things to change.
These options enable building an AArch64 capable buildroot using the Linaro AArch64 toolchain, with it outputting a terminal to ttyFIQ0 which is the headphone UART terminal. Along with taking the filesystem and pushing it in to a cpio archive which later the linux kernel understands how to use.
Target options->Target Architecture->AArch64
Toolchain->Toolchain type->External toolchain
External toolchain automatically enables the Linaro AArch64 toolchain below it.
System configuration->getty options->TTY port->ttyFIQ0
Filesystem images->cpio the root filesystem
3)*Build the buildroot
Code:
make
This will go through downloading all the packages required to build an AArch64 buildroot. So go make a cup of tea, this should only take a few minutes depending on your internet speed and computer speed.
Once the system has gone through compiling everything, then you'll have a file available.
This file will end up in the `output/images/` folder as the file 'rootfs.cpio'. If that file isn't there then something terrible has happened.
If the file is there, then everything on this step is done! We built our temporary filesystem which we will be putting to use in the next step!
This is really the meat of the guide. Building the kernel correctly is half the battle with running a full Linux distribution on the device. We are going to be pulling our kernel from the official Android repository and changing the default configuration to suit our needs.
Steps Overview
Clone kernel with git
Configure environment for cross-compiling
Configure the kernel for building correctly
Test run kernel
1) Clone the kernel using git
This step will take quite a bit of time since it is fairly large.
Code:
git clone https://android.googlesource.com/kernel/tegra.git
cd tegra
2)*Checkout the correct branch
Code:
git checkout android-tegra-flounder-3.10-lollipop-release
3)*Set environment variables for cross-compiling the Linux kernel
Code:
export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-gnu-
4) Configure the kernel
Code:
make flounder_defconfig
make menuconfig
This will bring up a menu just like how the buildroot menu came up. We need to set a few options to make sure everything comes up correctly on the Nexus 9.
General setup->Cross-compiler tool prefix->aarch64-linux-gnu-
General setup->Initramfs source file(s)-><The cpio file that was built in the buildroot>
Mine was '/home/ryanh/work/N9_Kernel_Tutorial/buildroot/output/images/rootfs.cpio'
Device Drivers->Character devices->Virtual Terminal
Device Drivers->Generic Driver Options->Maintain a devtmpfs filesystem to mount at /dev
Device Drivers->Generic Driver Options->Automount devtmpfs at /dev...
Device Drivers->Network device support->Wireless LAN->Firmware path->/lib/firmware/fw_bcmdhd.bin
Device Drivers->Network device support->Wireless LAN->NVRAM path->/etc/wifi/bcmdhd.cal
Device Drivers->Character devices->Virtual terminal
Device Drivers->Watchdog Timer Support->Tegra watchdog (Disable it)
Device Drivers->Staging drivers->Android->Put the FIQ debugger into console mode by default
Device Drivers->Graphics support->Tegra Framebuffer driver
Device Drivers->Graphics support->Console display driver support->Framebuffer Console support
Device Drivers->Graphics support->Console display driver support->Map the console to the primary display device
Device Drivers->Graphics support->Console display driver support->Framebuffer Console Rotation
We've got to enable the tegra watchdog otherwise the device will reboot automatically every two minutes.
The few other things are needed for a sane Linux environment and also setting the initramfs file as our root filesystem.
5)*Compile the kernel
Code:
make -j5
If you're lucky it will build without error and you'll have a file in 'arch/arm64/boot/' named 'Image.gz-dtb' which is the final kernel file.
With this file we should be able to boot a kernel with a buildroot automatically mounted.
Test run the kernel!
This is where I recommend having your headphone UART cable, because this won't update the screen at all and it will seem like it has locked up. The cable is fairly easy to make with a little bit of know how about soldering and electronics, and will save you from tearing out your hair trying to figure out what went wrong.
Without the cable you won't be able to see any output, so either build the cable or skip this step.
To see output from the cable you need to connect to it with either gnu screen or minicom
Code:
sudo screen /dev/ttyUSB0 115200
To boot the kernel reboot in to the bootloader and then boot the kernel using fastboot
Code:
adb reboot bootloader
fastboot -c "console=ttyFIQ0,115200n8 rw" boot arch/arm64/boot/Image.gz-dtb
Once you do this, over the UART cable you'll see the Linux kernel booting and have a buildroot login terminal.
The default username is 'root' without any password.
Code:
Welcome to Buildroot
buildroot login: root
# uname -a
Linux buildroot 3.10.40-ga3846f1 #2 SMP PREEMPT Sat Dec 27 06:15:13 CST 2014 aarch64 GNU/Linux
This shows that the kernel is booting properly and jumping in to our buildroot correctly.
To go back in to Android from this area just type reboot in to the terminal instance.
The final step in getting a Linux distro is to build our root filesystem on a USB flash drive.
Steps Overview
Format flash drive as ext2/ext3/ext4
Dump Ubuntu rootfs on to the flash drive
Minor configuration
Rebuild Linux kernel with new options
The first thing we will want to be doing is formatting a flash drive to a Linux partition type. I am going to be using an ext3 partition on a 64GB USB 3.0 flash drive. I'd recommend at least an 8GB flash drive, anything smaller may have issues with fitting everything on to it.
Once you're done flashing your drive you'll need to download an Ubuntu Core image for AArch64/ARM64/ARMv8.
Download the Ubuntu Core 14.04.1 image*Here. Make sure to grab the 'arm64' tar.gz file, not the 'amd64' file.
Once you have the flash drive formatted and mounted, extract the ubuntu core image to the flash drive.
Code:
sudo tar zxf ubuntu-core-14.04.1-core-arm64.tar.gz -C Mount/
Configuring the Ubuntu Core image for the Nexus 9
Steps Overview
Terminal over UART configuration
DNS configuration
Firmware files
Add user
In order to properly get a terminal instance over the UART we have to add a package to the base Ubuntu core image.
This is a small package that all it does is open a new terminal instance over the configured getty instance. The package can be found*Here.
To install it to the root filesystem just unpack it to the root filesystem on the USB flash drive
Code:
sudo tar xf console.tar -C Mount/
We need to set up a DNS server so that the filesystem will be able to resolve addresses via DNS.
Let's just set it to Google's DNS.
Code:
sudo echo 'nameserver 8.8.8.8' >> Mount/etc/resolv.conf
We've got to grab the firmware files from the Nexus 9 in order for the device to stop spamming warnings at us in the console.
These are used for multiple things, so it is a good idea to grab them. You can grab these either from a factory image or directly from the Nexus 9. I chose to grab mine directly from the Nexus 9.
Code:
sudo mkdir Mount/lib/firmware
sudo mkdir Mount/etc/wifi
sudo adb pull /vendor/firmware Mount/lib/firmware/
sudo adb pull /system/etc/wifi/bcmdhd.cal Mount/etc/wifi/
We need to add a user to the root filesystem. This is a fairly annoying step because we actually need to chroot in to the filesystem.
There is a really nice guide to doing this on a ARMv7 filesystem*Here. This won't work for our system because we are working with ARMv8 instead.
So we are going to use that guide as a base but change it over to support what we need to do for AArch64.
First thing we've got to do is build qemu as a static binary for AArch64
This is fairly straight forward.
Code:
git clone git://git.qemu-project.org/qemu.git
cd qemu
sudo apt-get build-dep qemu
./configure --target-list=aarch64-linux-user *--static --disable-werror
make -j5
This will get us a binary in the aarch64-linux-user folder called 'qemu-aarch64'
We will need to rename this to 'qemu-arm64-static' and move it in to the '/usr/bin/' folder inside of our root partition
Once qemu is inside of the root partition, we will be able to chroot in to it and add our user.
So go in to the root directory of our filesystem we are generating, and run a few basic commands.
Set up some mounts inside of the chroot
Code:
for m in `echo 'sys dev proc'`; do sudo mount /$m ./$m -o bind; done
*Chroot in to the root filesystem
Code:
sudo LC_ALL=C chroot . /bin/bash
Now we are inside of the root filesystem, we can add the new user to it.
Let's just add a new user named 'ubuntu'. The first command will ask for a password for your user. The rest will add it to some default groups to make sure it can do things.
Code:
adduser ubuntu
addgroup ubuntu adm
addgroup ubuntu sudo
Once the user is added you can exit the root filesystem with a regular exit command, then we have to make sure to unmount all of the mounts we did prior to chrooting in to the filesystem.
Code:
for m in `echo 'sys dev proc'`; do sudo umount ./$m; done
Make sure to cleanly unmount the flash drive so everything is written to it.
Reconfigure the kernel to boot from flash drive instead of initramfs
1)*Go in to the kernel configuration
Code:
make menuconfig
Change the configuration to remove the initramfs
General Setup->Initial RAM filesystem and RAM disk (initramfs/initrd) support (Disable it)
Then exit the menu and rebuild the kernel
Code:
make -j5
Running the Ubuntu Core Image
So with the device sitting at the bootloader we will need to boot the new kernel
Code:
fastboot -c "fbcon=rotate:1 root=/dev/sda1 rootwait rw" boot arch/arm64/boot/Image.gz-dtb
This will boot the kernel and then it'll wait until you plug in the flash drive to continue booting.
So plug the USB hub in to the USB OTG cable. Then plug your USB flash drive and USB keyboard in to the USB hub.
After that plug the USB OTG cable in to the tablet and it will continue booting.
Give it roughly 20-60 seconds and it will show a login prompt on the devices screen.
You'll be able to login to the core image with the user you created earlier.
With some packages installed like the xubuntu-desktop environment you can have a full desktop available.
Reserved for QA - Additional Information - Misc
Information
Hardware acceleration
Potentially hardware acceleration should be possible using the Nouveau video driver. This is completely untested, and the Nouveau version included with Ubuntu 14.04 isn't new enough to have GK20a support. One would most likely need to build Nouveau from ToT to get support.
Known Issues
If something tries enabling the wifi chipset then the kernel will hardlock and cause the device to reboot. It may be best to disable the broadcom driver in the kernel until that is figured out.
Bluetooth doesn't seem to work. Not sure why not. Seems to show up as a device, but it can't be used.
Wow thanks for taking the time to post this tuturial.
Greatly apriciated.
Nice work, excellent writeup.
Cant wait to try!!
---------- Post added at 08:35 PM ---------- Previous post was at 08:31 PM ----------
Where did you get your headphone UART cable
Great work sonic, been waiting for this. Will try and report back.
I took some time to figure out how to fix the integrated wireless LAN in the tablet.
This requires grabbing an additional file from the tablet and configuring a few kernel options so they know where the new file locations are since previously they are configured to Android specific locations.
Seems to work fine on my 802.11n 2.4Ghz network.
Running into a snag while compiling the kernel. It always fails at this exact spot on Ubuntu 14.10. Any ideas? (first time compiling a kernel from source )
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
farmerbb said:
Running into a snag while compiling the kernel. It always fails at this exact spot on Ubuntu 14.10. Any ideas? (first time compiling a kernel from source )
Click to expand...
Click to collapse
Note: that does not tell me what failed
When the build failes can you do
Code:
make
Then give me the output
USBhost said:
Note: that does not tell me what failed
When the build failes can you do
Code:
make
Then give me the output
Click to expand...
Click to collapse
OK, here's what failed. Looks like an error in qcom_usb_modem_power.c:
Looks like it's having a fit in a required module that is enabled for the tegra platform.
Looks like a new warning that has cropped up in a newer version of GCC? My AArch64 gcc version that I'm running is 4.8.2. I can install 4.9.1 and test there but it will have to be later.
GCC is definitely right in this case though, Qualcomm managed to screw up a basic memset function call. The zero is supposed to be the middle argument, and GCC recognizes it as an warning now. Which gets upgraded to an error with some configuration.
If you know light C/C++ you can change those two lines that it is complaining about to be correct.
Should be
memset(modem->ftrace_cmd, 0, sizeof(modem->ftrace_cmd));
and
memset(modem->msr_info_list, 0, sizeof(modem->msr_info_list));
For those two lines at 1912 and 1915 respectively.
Looks like I'm going to have to start a patch set for this to work around Qualcomm failing.
sonicadvance1 said:
Looks like it's having a fit in a required module that is enabled for the tegra platform.
Looks like a new warning that has cropped up in a newer version of GCC? My AArch64 gcc version that I'm running is 4.8.2. I can install 4.9.1 and test there but it will have to be later.
GCC is definitely right in this case though, Qualcomm managed to screw up a basic memset function call. The zero is supposed to be the middle argument, and GCC recognizes it as an warning now. Which gets upgraded to an error with some configuration.
If you know light C/C++ you can change those two lines that it is complaining about to be correct.
Should be
memset(modem->ftrace_cmd, 0, sizeof(modem->ftrace_cmd));
and
memset(modem->msr_info_list, 0, sizeof(modem->msr_info_list));
For those two lines at 1912 and 1915 respectively.
Looks like I'm going to have to start a patch set for this to work around Qualcomm failing.
Click to expand...
Click to collapse
I made the changes to qcom_usb_modem_power.c as you listed, and got the kernel to compile okay.
Starting from a fresh Ubuntu 14.10 install (inside a VM), I ended up installing these additional packages in addition to the ones you list in step 0:
Code:
libncurses5-dev libc6:i386 libstdc++6:i386 zlib1g:i386 qemu binfmt-support qemu-user-static android-tools-adb android-tools-fastboot
To get the kernel to compile, I also needed to symlink aarch64-linux-gnu-gcc-4.9 to aarch64-linux-gnu-gcc in /usr/bin, for make to recognize the correct gcc program.
But anyway, I've gotten my N9 booted to the Ubuntu login prompt Now I just need to grab a USB keyboard so that I can actually login!
Thanks so much for the guide! Can't wait for MultiROM to be ported to our device, to make this easier to boot
farmerbb said:
I made the changes to qcom_usb_modem_power.c as you listed, and got the kernel to compile okay.
Starting from a fresh Ubuntu 14.10 install (inside a VM), I ended up installing these additional packages in addition to the ones you list in step 0:
Code:
libncurses5-dev libc6:i386 libstdc++6:i386 zlib1g:i386 qemu binfmt-support qemu-user-static android-tools-adb android-tools-fastboot
To get the kernel to compile, I also needed to symlink aarch64-linux-gnu-gcc-4.9 to aarch64-linux-gnu-gcc in /usr/bin, for make to recognize the correct gcc program.
But anyway, I've gotten my N9 booted to the Ubuntu login prompt Now I just need to grab a USB keyboard so that I can actually login!
Thanks so much for the guide! Can't wait for MultiROM to be ported to our device, to make this easier to boot
Click to expand...
Click to collapse
ii cant wait
Excellent guide. Thank you very much.
It appears the "*" in this configure command below may be a typo?
sonicadvance1 said:
Code:
git clone git://git.qemu-project.org/qemu.git
cd qemu
sudo apt-get build-dep qemu
./configure --target-list=aarch64-linux-user *--static --disable-werror
make -j5
Click to expand...
Click to collapse
Anyway, I am running into a more serious issue (Problem resolved. See bottom.) executing dynamically build aarch64 binaries. On my 14.10 x86 machine, I am able to execute statically build aarch64 binaries using my new qemu-aarch64 emulator. I am not able to execute dynamically build aarch64 binaries. For instance, when I try to execute bin/bash from the mounted root filesystem directly on the host, it will complain:
/lib/ld-linux-aarch64.so.1: No such file or directory
Now this is all right, because I don't have the required aarch64 shared libs on my x86 machine. So I copied qemu-aarch64 to /usr/bin/qemu-aarch64-static on the mounted root filesystem. However, when I run chroot (executing the same /bin/bash):
sonicadvance1 said:
Code:
sudo LC_ALL=C chroot . /bin/bash
Now we are inside of the root filesystem, we can add the new user to it.
Click to expand...
Click to collapse
...I am getting this:
chroot: failed to run command ‘/bin/bash’: No such file or directory
I think this is due to the same issue as above: that qemu is looking for aarch64 shared libs (/lib/ld-linux-aarch64.so.1 is needed to run bash) on my host machine. Should it not, due to chroot, look for them in the mounted root filesystem (where they actually are)?
Other than this... my ARM64 kernel is booting fine. The only problem is, that I cannot login to it yet, because I wasn't able to add a new user.
Edit: Interestingly, the following "works":
Code:
$ sudo chroot . usr/bin/qemu-aarch64-static /bin/bash
bash: /usr/bin/groups: No such file or directory
[email protected]:/#
Now inside the chroot running the aarch64 variant of /bin/bash, I am still unable to start any aarch64 executables. But tab-completion works and I am able to find and run /usr/bin/qemu-aarch64-static (the x86 binary). Could this be a problem with any binfmts mappings inside chroot? Odd.
Edit 2: Solved. Just tried from another machine (10.04) and here, everything is working perfectly. I probably have chroot hosed on my 1st machine. Excellent guide. One last note: if you don't have qemu installed on your host machine, you may need to run the following:
Code:
sudo apt-get install qemu binfmt-support qemu-user-static
t1mur said:
Excellent guide. Thank you very much.
It appears the "*" in this configure command below may be a typo?
Anyway, I am running into a more serious issue executing dynamically build aarch64 binaries. On my 14.10 x86 machine, I am able to execute statically build aarch64 binaries using my new qemu-aarch64 emulator. I am not able to execute dynamically build aarch64 binaries. For instance, when I try to execute bin/bash from the mounted root filesystem directly on the host, it will complain:
/lib/ld-linux-aarch64.so.1: No such file or directory
Now this is all right, because I don't have the required aarch64 shared libs on my x86 machine. So I copied qemu-aarch64 to /usr/bin/qemu-aarch64-static on the mounted root filesystem. However, when I run chroot (executing the same /bin/bash):
...I am getting this:
chroot: failed to run command ‘/bin/bash’: No such file or directory
I think this is due to the same issue as above: that qemu is looking for aarch64 shared libs (/lib/ld-linux-aarch64.so.1 is needed to run bash) on my host machine. Should it not, due to chroot, look for them in the mounted root filesystem (where they actually are)?
Other than this... my ARM64 kernel is booting fine. The only problem is, that I cannot login to it yet, because I wasn't able to add a new user.
Edit: Interestingly, the following "works":
Code:
$ sudo chroot . usr/bin/qemu-aarch64-static /bin/bash
bash: /usr/bin/groups: No such file or directory
[email protected]:/#
Now inside the chroot running the aarch64 variant of /bin/bash, I am still unable to start any aarch64 executables. But tab-completion works and I am able to find and run /usr/bin/qemu-aarch64-static (the x86 binary). Could this be a problem with any binfmts mappings inside chroot? Odd.
Click to expand...
Click to collapse
I ran into these exact issues as well, and was eventually able to get the chroot working properly
Make sure you have the qemu, binfmt-support, qemu-user-static packages installed, and run 'update-binfmts --display' to ensure that aarch64 support is registered properly. Also make sure you've named the 'qemu-aarch64' binary as 'qemu-arm64-static' (not 'qemu-aarch64-static') at usr/bin on the flash drive (the Ubuntu Core image recognizes its architecture as arm64, not aarch64)
farmerbb said:
I ran into these exact issues as well, and was eventually able to get the chroot working properly
Make sure you have the qemu, binfmt-support, qemu-user-static packages installed, and run 'update-binfmts --display' to ensure that aarch64 support is registered properly. Also make sure you've named the 'qemu-aarch64' binary as 'qemu-arm64-static' (not 'qemu-aarch64-static') at usr/bin on the flash drive (the Ubuntu Core image recognizes its architecture as arm64, not aarch64)
Click to expand...
Click to collapse
Ah... it's really good to know one isn't alone. Hehe. Well, it looks like 14.10 is better suited for compiling the arm kernel (gcc-4.9-aarch64 etc.), but doing the chroot is smoother on 14.04.
But how do I get wifi and/or USB ethernet up? And why do have to look at this?
unable to stat /etc/sudoers: no such file or directory
Did you get this too?
t1mur said:
Ah... it's really good to know one isn't alone. Hehe. Well, it looks like 14.10 is better suited for compiling the arm kernel (gcc-4.9-aarch64 etc.), but doing the chroot is smoother on 14.04.
But how do I get wifi and/or USB ethernet up? And why do have to look at this?
unable to stat /etc/sudoers: no such file or directory
Did you get this too?
Click to expand...
Click to collapse
I'm still working on getting networking and X.org running myself. It will recognize my ASIX USB adapter (in the dmesg log) but eth0 doesn't show up and 'ifconfig eth0 up' doesn't work. Haven't tried Wi-Fi yet. My low-level Linux knowledge isn't all that great However, I can still chroot back into the filesystem on my PC and update/install packages, etc. in the meantime.
No I don't see the /etc/sudoers error.
Hi,
This is a guide on compiling the latest (as of 2015-06-24) dropbear SSH daemon (2015.67) for the Nexus 7 2013 but should also work for other ARM architecture Android devices. The guide is mainly based off the work from the blog http://blog.xulforum.org/index.php?post/2013/12/19/Compiling-Dropbear-for-a-Nexus-7-tablet which is an excellent guide to getting an older version of dropbear running on ARM Android devices.
I've modified the patch made by the original author to work on the latest dropbear version. There are not much changes from ver 2013.58 to 2015.67. The main difference seems to be the support for Elliptical Curve encryption.
** START DISCLAIMER **
I did not write the original codes/patch myself and have not scrutinized it for any security issues. USE IT AT YOUR OWN RISK.
** END DISCLAIMER **
Here's the list of requirements :-
1) Dropbear 2015.67 source code - dropbear-2015.67.tar.bz2 (https://matt.ucc.asn.au/dropbear/releases/dropbear-2015.67.tar.bz2)
2) Patch to compile for Android - dropbear-v67-android-patch-20150630 (https://goo.gl/Obo6kT) OR alternatively, you can use the patch from user serasihay (https://goo.gl/ip6Tkb).
3) A recent Linux distro (I use Ubuntu 14.04.2 LTS).
4) Development tools i.e. C compiler, linker, etc... all the necessary tools to run configure, make, etc..
5) Android NDK (I'm using rev 10e) installed & setup on Linux.
Steps :-
(1) Extract the source code to a directory of its own :-
Code:
tar xjf dropbear-2015.67.tar.bz2
cd dropbear-2015.67
(2) Patch the source :-
Code:
patch -p1 < dropbear-v67-android-patch-20150630
(3) Run configure :-
Code:
./configure --build=x86_64-unknown-linux-gnu --host=arm-linux-androideabi \
--disable-zlib --disable-largefile --disable-loginfunc \
--disable-shadow --disable-utmp --disable-utmpx --disable-wtmp \
--disable-wtmpx --disable-pututline --disable-pututxline --disable-lastlog
(4) Edit config.h, search for USE_DEV_PTMX and add the following line after that :-
Code:
#define USE_DEV_PTMX 1
(5) Run make :-
Code:
STATIC=1 MULTI=1 SCPPROGRESS=0 PROGRAMS="dropbear dropbearkey scp dbclient" make strip
(6) You should end up with a single static binary "dropbearmulti" which you should link dropbear, dbclient/ssh, dropbearkey and scp to.
Code:
./dropbear -h
Dropbear server v2015.67 https://matt.ucc.asn.au/dropbear/dropbear.html
Usage: ./dropbear [options]
-A Android Mode, specify a user explicitly
-N Android Mode, user name
-C Android Mode, password
-R Android Mode, public key file (authorized_keys)
-U Android Mode, UID
-G Android Mode, GID
-b bannerfile Display the contents of bannerfile before user login
(default: none)
-r keyfile Specify hostkeys (repeatable)
defaults:
dss /etc/dropbear/dropbear_dss_host_key
rsa /etc/dropbear/dropbear_rsa_host_key
ecdsa /etc/dropbear/dropbear_ecdsa_host_key
-F Don't fork into background
-E Log to stderr rather than syslog
-m Don't display the motd on login
-w Disallow root logins
-s Disable password logins
-g Disable password logins for root
-B Allow blank password logins
-j Disable local port forwarding
-k Disable remote port forwarding
-a Allow connections to forwarded ports from any host
-p [address:]port
Listen on specified tcp port (and optionally address),
up to 10 can be specified
(default port is 22 if none specified)
-P PidFile Create pid file PidFile
(default /var/run/dropbear.pid)
-i Start for inetd
-W <receive_window_buffer> (default 24576, larger may be faster, max 1MB)
-K <keepalive> (0 is never, default 0, in seconds)
-I <idle_timeout> (0 is never, default 0, in seconds)
-V Version
I will assume you know what to do with the binary file generated so will not elaborate on the process.
Hope it works for you guys. I'll be happy to help out anyone who needs more details.
NOTE: I have posted a similar guide for building dropbear for an x86 Android device in the Zenfone 2 forum (http://forum.xda-developers.com/zenfone2/general/compiling-dropbear-2015-67-zenfone-2-t3142222). The steps are mostly similar.
Cheers.
UPDATE: 2015-06-29
- I've updated the patch to fix dbclient/ssh client using password authentication.
UPDATE: 2015-06-30
- Reverted the patch for the dbclient fix as NDK does not have getpass() function. Looks like dbclient with password auth will seg fault until we can find an alternative to getpass().
UPDATE: 2015-06-30 (2)
- Uploaded yet another update to the patch to include a version of getpass(). Hopefully it works
UPDATE: 2015-07-02
- Added alternate dropbear patch by serasihay (Thanks!). Details of changes can be found here :- http://forum.xda-developers.com/showpost.php?p=61649194&postcount=14
Thank you for the guide, wolfdude!
I'm more interested in the client rather than the server, though.
Have you tried it? Does it work for you? For me, it doesn't.
I'm tying to connect to a remote host using the password authentication, but a segmentation fault occurs just after accepting the key of the remote host. It doesn't get to ask for the password.
The crosscompilation process goes without errors. I copy the binary to the Nexus, I do the symlinks in /system/xbin, etc.
I run the client from Terminal Emulator, and the remote host key is correctly stored in .ssh/known_hosts, inside the app's "app_HOME" directory found in /data.
On the remote server I can see that the connection request is received, the provided user is successfully identified as valid/existing; and the key algorithm exchange seems to succeed, too... But after that the connection is closed by the client... :-?
Hi serasihay,
Yes, I'm getting a Seg Fault too when I use the SSH client and using password auth. When I use an identity file, I don't get a SegFault and it connects successfully. I'm not sure what's causing the Seg Fault. When I get some time, I might have to run it through a debugger to find out. There might be more patches required to get the client working.
Cheers.
Thank you for confirming it, wolfdude.
I also think that there's more code to be patched.
I'm reading to learn how to debug the binary... Never done this on Android...
Thanks again for your time!
Hi Serasihay,
I've updated the patch to fix the seg fault issue. Turns out that the original patch commented out a chunk of code that was required for password authentication. Please test it out and hopefully this works well for you.
Cheers.
It looks like you've uploaded the old patch again by mistake... I diffed both files and they are exactly the same.
Yesterday I made good progress learning how to debug remotely a binary running on an Android device. I had to recompile the binary without striping the symbols, etc.
I'm a bit out of my depth here, but I'll continue trying to understand where's the flaw.
Code:
$ adb forward tcp:5039 tcp:5039
$ ./android-ndk-r10e/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gdb
[...]
(gdb) target remote :5039
(gdb) symbol-file dropbear-2015.67/dropbearmulti
Reading symbols from dropbear-2015.67/dropbearmulti...done.
(gdb) run
The "remote" target does not support "run". Try "help target" or "continue".
(gdb) continue
Continuing.
Program received signal SIGSEGV, Segmentation fault.
0x000527cc in strlen (s=0x0) at bionic/libc/arch-arm/bionic/strlen.c:62
62 bionic/libc/arch-arm/bionic/strlen.c: No such file or directory.
(gdb) backtrace
#0 0x000527cc in strlen (s=0x0) at bionic/libc/arch-arm/bionic/strlen.c:62
#1 0x0000aff0 in cli_auth_password ()
#2 0x0000a7ec in cli_auth_try ()
#3 0x0000f534 in cli_sessionloop ()
#4 0x000161a0 in session_loop ()
#5 0x0000f16c in cli_session ()
#6 0x0000d604 in cli_main ()
#7 0x00008228 in main ()
(gdb) frame 1
#1 0x0000aff0 in cli_auth_password ()
(gdb) info frame
Stack level 1, frame at 0xbecbf848:
pc = 0xaff0 in cli_auth_password; saved pc = 0xa7ec
called by frame at 0xbecbf860, caller of frame at 0xbecbf7d0
Arglist at 0xbecbf844, args:
Locals at 0xbecbf844, Previous frame's sp is 0xbecbf848
Saved registers:
r4 at 0xbecbf834, r5 at 0xbecbf838, r6 at 0xbecbf83c, r11 at 0xbecbf840, lr at 0xbecbf844
(gdb)
I guess the code commented out in the patch that you are referring to is the one that, in cli-auth.c, does the following aasignement:
Code:
password = getpass(prompt);
It makes sense, because from the debugging "session" I posted int my previous post, one could deduce that the problem is that a null string is passed to cli_auth_password ()...
But, unhappily, the function getpass is not provided by the Android C library (Bionic). If you try to build with a "corrected" patch, you end up with this error:
Code:
cli-auth.o:cli-auth.c:function getpass_or_cancel: error: undefined reference to 'getpass'
I'll try to find a workaround. Any help will be appreciated!
serasihay said:
I guess the code commented out in the patch that you are referring to is the one that, in cli-auth.c, does the following aasignement:
...
I'll try to find a workaround. Any help will be appreciated!
Click to expand...
Click to collapse
Hi serasihay,
You're on the right track That is indeed the function causing the seg fault. I've updated the post to have the correct URL to the CORRECT patch now. Please try again and let me know how it goes.
Cheers.
Hi wolfdude,
The corrected patch you've uploaded is like the one I already tried. As I was saying in my previous post:
serasihay said:
But, unhappily, the function getpass is not provided by the Android C library (Bionic). If you try to build with a "corrected" patch, you end up with this error:
Code:
cli-auth.o:cli-auth.c:function getpass_or_cancel: error: undefined reference to 'getpass'
Click to expand...
Click to collapse
So, in my opinion, it's a matter of finding an alternative to "getpass", or implementing a new one... I'm reading what would be the better solution.
Thank you again for your time!
Cheers.
Ahh... apologies for not reading correctly. I see what you meant now. I did my compile/build on an x86 android device and I assumed that it would also work for ARM using the NDK. Let us know how you go with the getpass() alternative.
Thanks.
Hi serasihay,
Ok... I've uploaded another patch this time to include a getpass() function which I pinched & modified from the GLIBC library. Hopefully it works as expected. Again... I didn't audit it for any security issues so use at your own risk
Cheers.
Thank you very much for your help and efforts, wolfdude.
I've taken my chances with your new patch... But the code doesn't compile, either.
Now the problem is with the "getline" function:
Code:
cli-auth.o:cli-auth.c:function getpass: error: undefined reference to 'getline'
I'll keep struggling with it as time allows...
Edit:
I've found a working getpass function. The code compiles and allows me to successfuly login to my remote machine!
https://github.com/CyanogenMod/android_external_dropbear/blob/cm-12.0/netbsd_getpass.c
When I get time I'll see if it is secure to include it and what would be the proper way to do so. I'm out of my depth here. Of course, feel free to make a new patch or whatever you consider appropriate, surely you're more experienced than I am!
Good to know you have it working now. I'm no developer too so can't really comment on which ever is better. As long as it works right
Cheers.
I've uploaded the patch that makes the dropbear client work for me.
https://goo.gl/ip6Tkb
Basically, it is the one you uploaded initially, but it corrects the problem with the getpass function by using the one implemented in the file netbsd_getpass.c.
To sum it up, my patch differs from your first one in that:
* Adds two new files, netbsd_getpass.h and netbsd_getpass.c.
* Modifies cli-auth.c to include the netbsd_getpass.h in order to use its getpass function.
* Modifies the Makefile.in so that the netbsd_getpass.c object is compiled and included into the dropbearmulti binary.
* Modifies options.h to change the path of the SSH which is used in scp.c to: _PATH_SSH_PROGRAM "/system/xbin/dbclient"; i.e.: it assumes that you do the appropiate symlink in /system/xbin...
Cheers!
Good stuff! I'll update the original post to have a link to your patch.
Thanks.
I'll be happy if it helps anyone.
Thank you!
If you have openssl ported to Android, there's a simple, useful patch that substitutes crypt with openssl's DES_crypt.
Link to the description (which contains a link to the patch) is here: http://www.gossamer-threads.com/lists/openssh/bugs/55836
Note: DES encryption isn't the best, but it's better than cleartext. I did look through the openssl headers/libs for an AES_crypt or 3DES_crypt but couldn;t find anything
static compile... 'No such file or directory" error
Excellent guide -- wolfdude & serasihay thank-you. I'm new to andriod and was clueless on how to do this.
I have the dbclient running, but only if I compile without the STATIC=1. Whenever I use static I get an "sh: ./dbclient: No such file or directory" error immediately upon execution. Any idea why this is happening?
I'm using a Motorola Moto G (the new Moto G sometimes called the Moto G2). As the non-static is quite a bit smaller and seems to run okay I'm wondering if not using static will bomb on other devices and I'm just lucky with the Moto G.
Hi,
No idea why you can't compile it as static... but if the dynamic executable works for you then it should be ok.
Cheers.
I have tried to compile dropbear with both patches. But everytime it shows ::
gcc -I./libtomcrypt/src/headers/ -I. -I. -Os -W -Wall -Wno-pointer-sign -DDROPBEAR_SERVER -DDROPBEAR_CLIENT -DDBMULTI_dropbear -DDBMULTI_dbclient -DDBMULTI_dropbearkey -DDBMULTI_dropbearconvert -DDBMULTI_scp -DDROPBEAR_MULTI -c -o netbsd_getpass.o netbsd_getpass.c
netbsd_getpass.c: In function 'getpass':
netbsd_getpass.c:68:18: error: '_PASSWORD_LEN' undeclared (first use in this function)
netbsd_getpass.c:68:18: note: each undeclared identifier is reported only once for each function it appears in
netbsd_getpass.c:68:14: warning: unused variable 'buf' [-Wunused-variable]
netbsd_getpass.c:114:1: warning: control reaches end of non-void function [-Wreturn-type]
make: *** [netbsd_getpass.o] Error 1
Click to expand...
Click to collapse
What is this tutorial?
This tutorial will:
Creating an unofficial build of LineageOS 18.1 suitable for using to re-lock the bootloader on a OnePlus 8t
Take you through the process of re-locking your bootloader after installing the above
This tutorial will NOT:
Remove *all* warning messages during boot (the yellow "Custom OS" message will be present though the orange "Unlocked bootloader" message will not)
Allow you to use official builds of LineageOS 18.1 on your device with a re-locked bootloader (more details near the end of the tutorial)
This tutorial will assume you are working on an Ubuntu 18.04 installation, if you are using Windows or another Linux distro, the commands may be different.
Supported devices:
The following devices have been tested and confirmed to work:
OnePlus 7 Pro (guacamole)
OnePlus 8t (kebab)
Pixel 4 (flame)
Other OnePlus devices that support AVBv2 (OnePlus 6t and newer as well as most Pixel devices) and LineageOS 18.1 (see current support list over on the LineageOS download page) should work as well.
For simplicities sake, all further references will only be to the 8t (kebab).
Pre-requisites:
a mid level knowledge of terminal commands and features
a supported phone
a PC with enough CPU/RAM to build LineageOS 18.1 (recommended 8 cores, 24g of RAM)
a working USB cable
fastboot/adb installed and functional
LineageOS 18.1 source code downloaded
at least one successful build of LineageOS
at least one successful signing of your build with your own keys
Misc. notes:
the basics of building/signing of LineageOS is outside the scope of this tutorial, refer to the LineageOS Wiki for details on how to complete these tasks
you'll be modifying some code in LineageOS, so if you are not comfortable using basic editing utilities as well as patch, do not proceed any further
the path to your LineageOS source code is going to be assumed to be ~/android/lineageos, if it is somewhere else, substitute the correct path in the tutorial
the path to your private certificate files is going to be assumed to be ~/android-certs, if it is somewhere else, substitute the correct path in the tutorial
*** WARNING ****
This process may brick your device. Do not proceed unless you are comfortable taking this risk.
*** WARNING ****
This process will delete all data on your phone! Do not proceed unless you have backed up your data!
*** WARNING ****
Make sure you have read through this entire process at least once before attempting, if you are uncomfortable with any steps include in this guide, do not continue.
And now on with the show!
Step 1: Basic setup
You need a few places to store things, so create some working directories:
Code:
mkdir ~/android/kebab
mkdir ~/android/kebab/patches
mkdir ~/android/kebab/pkmd
You also need to add "~/android/lineageos/out/host/linux-x86/bin" to your shell's profile path. Make sure to close and restart your session afterwards otherwise the signing will fail later on with a "file not found" error message (this may no longer be required).
Step 2: Update kebab's BoardConfig.mk
You will need to add a few parameters to the end of ~/android/lineageos/device/oneplus/kebab/BoardConfig.mk, they are:
Code:
BOARD_AVB_ALGORITHM := SHA256_RSA2048
BOARD_AVB_KEY_PATH := /home/<userid>/.android-certs/releasekey.key
Note you cannot use "~" in the path names above to signify your home directory, so give the full absolute path to make sure the files are found.
Step 3: Update sm8250-common's BoardConfigCommon.mk
LineageOS by default disables Android Verified Boot's partition verification, but you can enable it now as all the required parts will be in place.
To enable partition verification do the following:
Code:
cd ~/android/lineageos/device/oneplus/sm8250-common
sed -i 's/^BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS += --flags 2/#BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS += --flags 2/' BoardConfigCommon.mk
sed -i 's/^BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS += --set_hashtree_disabled_flag/#BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS += --set_hashtree_disabled_flag/' BoardConfigCommon.mk
sed -i 's/^BOARD_AVB_VBMETA_SYSTEM_KEY_PATH := external\/avb\/test\/data\/testkey_rsa2048.pem/BOARD_AVB_KEY_PATH := \/home\/<userid>\/.android-certs\/releasekey.key/' BoardConfigCommon.mk
Don't forget to replace your <userid> in the third sed command above with your current logged in user id.
Step 4: Patch the AOSP and Device Makefile
You also need to patch the Makefile included with AOSP as it will otherwise fail during the build.
The required patch can be found here:
https://raw.githubusercontent.com/Wunderment/build_tasks/master/source/core-Makefile-fix-18.1.patch
Download it and store in ~/android/kebab/patches.
Now apply it with the following command:
Code:
cd ~/android/lineageos/build/core
patch Makefile ~/android/kebab/patches/core-Makefile-fix-18.1.patch
If you would like to know more about this patch, see the additional info at the bottom of this post.
There is also a small addition to the device's common.mk required to enable the OEM unlock option in developers options, do this via the following commands:
Code:
cd ~/android/lineageos/device/oneplus/sm8250-common
sed -i 's/^# OMX/# OEM Unlock reporting\nPRODUCT_DEFAULT_PROPERTY_OVERRIDES += \\\n ro.oem_unlock_supported=1\n\n# OMX/' common.mk
Step 5: Build LineageOS
You are now ready to build:
Code:
cd ~/android/lineageos
breakfast kebab
source build/envsetup.sh
croot
mka target-files-package otatools
Step 6: Sign the APKs
You are now ready to sign the apks with sign_target_files_apks:
Code:
./build/tools/releasetools/sign_target_files_apks -o -d ~/.android-certs $OUT/obj/PACKAGING/target_files_intermediates/*-target_files-*.zip signed-target_files.zip
Step 7: Build the OTA
Now it is time to complete the OTA package:
Code:
./build/tools/releasetools/ota_from_target_files -k ~/.android-certs/releasekey --block signed-target_files.zip lineage-18.1-[date]-UNOFFICIAL-kebab-signed.zip
Note, replace [date] with today's date in YYYYMMDD format.
Step 8: Create pkmd.bin for your phone
Before you can lock your phone, you have to tell it what your public key is so it knows it can trust your build.
To do this you need to create a pkmd.bin file:
Code:
~/android/lineageos/external/avb/avbtool extract_public_key --key ~/.android-certs/releasekey.key --output ~/android/kebab/pkmd/pkmd.bin
Step 9: Flashing your LineageOS build
It's time to flash your build to your phone. The following steps assume you have already unlocked your phone and have flashed an official version of LineageOS to it. You don't need to have flashed LineageOS yet, you could use TWRP through "fastboot boot" if you prefer. Or, if you want to use the recovery that was just created, it is located in ~/android/lineageos/out/target/product/kebab and is called recovery.img.
Reboot your phone in to recovery mode
In LineageOS Recovery return to the main menu and select "Apply update"
From your PC, run:
Code:
adb sideload ~/android/lineageos/lineage-18.1-[date]-UNOFFICIAL-kebab-signed.zip
When the sideload is complete, reboot in to LineageOS. Make sure everything looks good with your build.
You may also need to format your data partition at this time depending on what you had installed on your phone previously.
Step 10: Flashing your signing key
Now it's time to add your signing key to the Android Verified Boot process. To do so, do the following:
Reboot your phone in to fastboot mode
From your PC, run:
Code:
fastboot flash avb_custom_key ~/android/kebab/pkmd/pkmd.bin
fastboot reboot bootloader
fastboot oem lock
On your phone, confirm you want to re-lock and it will reboot
Your phone will then factory reset and then reboot in to LineageOS.
Which of course means you have to go through the first time setup wizard, so do so now.
Step 11: Disable OEM unlock
Congratulations! Your boot loader is now locked, but you can still unlock it again using fastboot, so it's time to disable that as well.
Unlock you phone and go to Settings->About phone
Scroll to the bottom and find "Build number"
Tap on it you enable the developer options
Go to Settings->System->Advanced->Developer options
Disable the "OEM unlocking" slider
Reboot
Step 12: Profit!
Other things
The above will build a standard USERDEBUG version of LineageOS, however this will still allow LineageOS Recovery to sideload non-signed files as well as give you root shell access through ADB. Step 3/4 above protects your system/vendor/boot/dtbo/etc. partitions, but none of the others. Likewise USERDEBUG builds will allow for rolling back to a previous builds/versions of LineageOS. To increase security and disallow both of these scenarios you may want to build a USER version of LineageOS to install. However this brings in other issues, such as flashing newer firmware from OnePlus so make sure you understand the implications of both choices. For more details on build types, see https://source.android.com/setup/develop/new-device#build-variants.
In the above example the releasekey from your LineageOS install has been used to sign AVB, but AVB supports other key strengths up to SHA512_RSA8192. You could create a key just for signing AVB that used different options than the default keys generated to sign LineageOS.
If you want to remove you signing key from your phone, you can do it by running "fastboot erase avb_custom_key".
The changes you made to the AOSP Makefile may conflict with future updates that you pull from LineageOS through repo sync, if you have to reset the file to get repo sync to complete successfully, you'll have to reapply the changes afterwards.
So why can't I do this with official LineageOS builds?
NEW: You can! See this thread for more details.
For Android Verified Boot (AVB) to work, it must have the hash values for each of the system/vendor/boot/dtbo/etc. partitions stored in vbmeta. Official LineageOS builds for kebab do include the vendor.img in them along with everything else that is needed, however that is not true for all phones.
There are two "issues" that stop someone from using the official kebab builds:
LineageOS does not provide a pkmd.bin file to flash to your phone to include the public key in your AVB process (NEW: this thread shows you how to extract the key).
AVB is enabled in the official LineageOS builds but does not validate the hash trees during boot which limits the protection offered.
Ok, what messages do I see during the boot process then?
During a boot you will of course see the standard OnePlus power up screen, followed by the yellow "custom os" message and then the standard LineageOS boot animation.
For more details on AVB boot messages, see https://source.android.com/security/verifiedboot/boot-flow
So what does that patch to the Makefile do?
AOSP's default Makefile makes an assumption that when AVB is enabled, that all the img files will be available well before vbmeta.img is created. This is simply NOT true and AOSP seems to know this as well from the following comment in the Makefile:
Code:
# Not using INSTALLED_VBMETA_SYSTEMIMAGE_TARGET as it won't be set yet.
ifdef BOARD_AVB_VBMETA_SYSTEM
$(eval $(call check-and-set-avb-args,vbmeta_system))
endif
ifdef BOARD_AVB_VBMETA_VENDOR
$(eval $(call check-and-set-avb-args,vbmeta_vendor))
endif
These two calls eventual evaluate to returning the path to the partitions based upon the INSTALLED_*IMAGE_TARGET variable, which isn't created until later in the build process.
Because of this, the command to build vbmeta.img gets corrupted due to the missing make variable being empty and an invalid command line is passed to avbtool near the end of the build.
The corruption happens due to the fact that the following line from the original Makefile:
Code:
--include_descriptors_from_image $(call images-for-partitions,$(1))))))
Gets added to the avbtool call even if "$(call images-for-partitions,$(1))" turns out to be an empty string. Avbtool then throws an error message as it is expecting a parameter after the "--include_descriptors_from_image" flag that is added for the "empty" partition path.
The fix is to call "$(call images-for-partitions,$(1))" earlier, set it to a variable and check to make sure it isn't an empty string before letting the "--include_descriptors_from_image" be added to the avbtool command line to be used later.
This technically generates an incomplete vbmeta.img file during the build process, but since the signing process recreates it from scratch anyway; no harm, no foul.
Thank You's
Obviously to all of the members of the LineageOS team!
LuK1337 for supporting kebab
optimumpro for the OnePlus 5/5t re-locking guide (https://forum.xda-developers.com/oneplus-5/how-to/guide-relock-bootloader-custom-rom-t3849299)which inspired this one
Quark.23 for helping with the process and testing on enchilada for my previous guide (https://forum.xda-developers.com/t/...s-6t-with-a-self-signed-build-of-los.4113743/) with the Oneplus 6/6t and LineageOS 17.1
Is root with magisk possibe with an locked bootloader? Would that require signing the magisk
-patched boot.img or packing magisk into the boot.img at build time?
coloneyescolon said:
Is root with magisk possibe with an locked bootloader? Would that require signing the magisk
-patched boot.img or packing magisk into the boot.img at build time?
Click to expand...
Click to collapse
You would have to include magisk in the build process, if you tried to "patch" the boot image after signing it would fail to boot as it would no longer have the right hash and you'd get the "currupt OS" message.
Is it possible signing the boot image after patching it with magisk?
Hello,
I followed the exact steps, and The build failed for OnePlus 7 Pro (guacamole), with this error:
error: device/oneplus/sm8150-common/fod/Android.bp:16:1: "[email protected]us_msmnile" depends on undefined module "//device/oneplus/
common:[email protected]"
error: device/oneplus/sm8150-common/fod/Android.bp:16:1: "[email protected]us_msmnile" depends on undefined module "//device/oneplus/
common:[email protected]"
16:07:07 soong bootstrap failed with: exit status 1
#### failed to build some targets (10 seconds) ####
ahmed.elsersi said:
Hello,
I followed the exact steps, and The build failed for OnePlus 7 Pro (guacamole), with this error:
error: device/oneplus/sm8150-common/fod/Android.bp:16:1: "[email protected]us_msmnile" depends on undefined module "//device/oneplus/
common:[email protected]"
error: device/oneplus/sm8150-common/fod/Android.bp:16:1: "[email protected]us_msmnile" depends on undefined module "//device/oneplus/
common:[email protected]"
16:07:07 soong bootstrap failed with: exit status 1
#### failed to build some targets (10 seconds) ####
Click to expand...
Click to collapse
That looks like you're missing some of the proprietery blobs, did you verify LineageOS comipled successfully before making any changes? Did you use the extract files script or use the muppets repo?
WhitbyGreg said:
That looks like you're missing some of the proprietery blobs, did you verify LineageOS comipled successfully before making any changes? Did you use the extract files script or use the muppets repo?
Click to expand...
Click to collapse
Hello,
I did extract the proprietary blobs from payload-based.
Do you mean I should compile LinageOS successfully first using:
source build/envsetup.sh
breakfast guacamole
croot
brunch guacamole
before i follow the steps listed here in this guide??
Thank You
ahmed.elsersi said:
Hello,
I did extract the proprietary blobs from payload-based.
Do you mean I should compile LinageOS successfully first using:
source build/envsetup.sh
breakfast guacamole
croot
brunch guacamole
before i follow the steps listed here in this guide??
Thank You
Click to expand...
Click to collapse
Check the extraction script for errors or switch to the muppets, sometimes the extraction script isn't up to date.
In general, yes, make sure you have a version of LineageOS that compiles successfully, that way you know you have a valid base to start from.
Pre-requisites:
at least one successful build of LineageOS
at least one successful signing of your build with your own keys
Click to expand...
Click to collapse
WhitbyGreg said:
Check the extraction script for errors or switch to the muppets, sometimes the extraction script isn't up to date.
In general, yes, make sure you have a version of LineageOS that compiles successfully, that way you know you have a valid base to start from.
Click to expand...
Click to collapse
Thank You so much.
One last question if i may, can these steps applied on LinageOS 4 MicroG using the automated build by their docker image docker-lineage-cicd ?
Thank You
ahmed.elsersi said:
Thank You so much.
One last question if i may, can these steps applied on LinageOS 4 MicroG using the automated build by their docker image docker-lineage-cicd ?
Thank You
Click to expand...
Click to collapse
You'd have to modify the docker image from my understanding as it includes all the source and tools required to do the build.
Hello,
Kindly Please, Could you clarify what do you mean by ~/.android-certs/releasekey.key and ~/.android-certs/releasekey.key/ ??
I created my own signing keys, and the output contains releasekey.pk8 and releasekey.x509.pem, that is why I'm confused.
Note: I did a successful build of LineageOS with OEM unlock support and its option show in development menu and I flashed it to my OnePlus 7 Pro, I used only that option:
cd ~/android/lineageos/device/oneplus/sm8250-common
sed -i 's/^# OMX/# OEM Unlock reporting\nPRODUCT_DEFAULT_PROPERTY_OVERRIDES += \\\n ro.oem_unlock_supported=1\n\n# OMX/' common.mk
Thank You
ahmed.elsersi said:
Hello,
Kindly Please, Could you clarify what do you mean by ~/.android-certs/releasekey.key and ~/.android-certs/releasekey.key/ ??
I created my own signing keys, and the output contains releasekey.pk8 and releasekey.x509.pem, that is why I'm confused.
Click to expand...
Click to collapse
You might need to convert your pk8 in to plain text using openssl like so:
openssl pkcs8 -in releasekey.pk8 -out releasekey.key
Click to expand...
Click to collapse
WhitbyGreg said:
You might need to convert your pk8 in to plain text using openssl like so:
Click to expand...
Click to collapse
Thank You for the help.
I'm sorry, it did not work, that's what i got:
Error reading key
139625476420992:error:0909006C:EM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: ENCRYPTED PRIVATE KEY
WhitbyGreg said:
You might need to convert your pk8 in to plain text using openssl like so:
Click to expand...
Click to collapse
I used the releasekey.x509.pem file, it is a PEM certificate text file, the build failed.
Hello,
Kindly please, clarify what is releasekey.key stands for, is it the private key or the public ? Is it data file or text file.
the build fail to the same.
avbtool extract_public_key --key ~/keys/releasekey.x509.pem --output ~/public_key.key
/android/lineageos/out/host/linux-x86/bin/avbtool: Error getting public key: unable to load Public Key
140081520305536:error:0909006C:EM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: PUBLIC KEY
avbtool extract_public_key --key ~/keys/releasekey.pk8 --output ~/public_key.key
/android/lineageos/out/host/linux-x86/bin/avbtool: Error getting public key: unable to load Public Key
140477081752960:error:0909006C:EM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: PUBLIC KEY
ahmed.elsersi said:
Thank You for the help.
I'm sorry, it did not work, that's what i got:
Error reading key
139625476420992:error:0909006C:EM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: ENCRYPTED PRIVATE KEY
Click to expand...
Click to collapse
What commandline did you use exactly?
ahmed.elsersi said:
I used the releasekey.x509.pem file, it is a PEM certificate text file, the build failed.
Click to expand...
Click to collapse
You can't use that.
WhitbyGreg said:
You can't use that.
Click to expand...
Click to collapse
I'm trying to understand, What is releasekey.key file??, it contains private key or public key, or both, and is it a data file or text file??
I did this:
openssl x509 -in releasekey.x509.pem -pubkey -out releasekey.key
The outputfile is a text and contains the public key and the certificate
when i delete the certificate part and start the build, i get this error:
/android/lineageos/out/host/linux-x86/bin/avbtool: Error signing: unable to load Private Key
140394811372928:error:0909006C:EM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: ANY PRIVATE KEY
if i delete the public key part, i get this error:
/android/lineageos/out/host/linux-x86/bin/avbtool: Error getting public key: unable to load Public Key
139655441114496:error:0909006C:EM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: PUBLIC KEY
if i didn't change anything and used the output file releasekey.key and start the build, i get this error:
/android/lineageos/out/host/linux-x86/bin/avbtool: Error signing: unable to load Private Key
139736685180288:error:0909006C:EM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: ANY PRIVATE KEY
I did a successful LineageOS signed build with my own generated keys and flashed on my mobile and working fine.
So, Kindly please, Could you please just tell us, What is this releasekey.key file, and how can we generate this releasekey.key ?
Thank You
ahmed.elsersi said:
when i delete the certificate part and start the build, i get this error:
Click to expand...
Click to collapse
Why did you delete anything?
ahmed.elsersi said:
So, Kindly please, Could you please just tell us, What is this releasekey.key file, and how can we generate this releasekey.key ?
Click to expand...
Click to collapse
releasekey.key is the plaintext private key for the release certificate.
WhitbyGreg said:
Why did you delete anything?
releasekey.key is the plaintext private key for the release certificate.
Click to expand...
Click to collapse
Following the LineageOS signing build steps, these files are generated:
media.pk8, networkstack.pk8, platform.pk8, releasekey.x509.pem, shared.x509.pem, testkey.x509.pem, media.x509.pem , networkstack.x509.pem , platform.x509.pem , releasekey.pk8, shared.pk8, testkey.pk8
I'm sorry, for the last 2 days I'm spinning around myself to figure out how to complete your guide and get a successful build.
Could you please, if you do not mind, just tell me how to generate this releasekey.key plaintext private key for the release certificate?
Your help is highly appreciated, thank you