DynamoRIO
i#2154 Make build for Android aarch64 possible.
Fix build issue from #2154 , but still not working on Android 9 AArch64.
Workaround trick: #3543
Files under ext/drsyms/libelftc-android-aarch64 is static library, for security, need to be built by Dynamorio team.
Thank you for the contribution! Our main AArch64 expert is out of office for a few days so please bear with a delay for a review.
Our AArch64 precommit test infrastructure was broken but is back up now. Please ensure it passes and I will review, thanks.
is there any issue with getting this merged? thanks!
The patch needs to be updated to the latest master and all the tests need to pass before it can be reviewed and merged.
Files under ext/drsyms/libelftc-android-aarch64 is static library, for security and trust, I won't build and upload them.
Docs also need to be update after merge. My steps to build:
# Download ndk, any ndk with gcc should be okay.
wget https://dl.google.com/android/repository/android-ndk-r14b-linux-x86_64.zip
# Uncompress
...
# Make standalone toolchains for android-23
android-ndk-r14b/build/tools/make-standalone-toolchain.sh --arch=arm64 --platform=android-23 --install-dir=/PATH/TO/STANDALONE/TOOLCHAINS --toolchain=aarch64-linux-androideabi-4.9
# Configure with cmake
cmake -DCMAKE_TOOLCHAIN_FILE=../make/toolchain-android-arm64.cmake -DANDROID_TOOLCHAIN=/PATH/TO/STANDALONE/TOOLCHAINS -DDR_COPY_TO_DEVICE=OFF
Files under ext/drsyms/libelftc-android-aarch64 is static library, for security and trust, I won't build and upload them.
Docs also need to be update after merge. My steps to build:
# Download ndk, any ndk with gcc should be okay. wget https://dl.google.com/android/repository/android-ndk-r14b-linux-x86_64.zip # Uncompress ... # Make standalone toolchains for android-23 android-ndk-r14b/build/tools/make-standalone-toolchain.sh --arch=arm64 --platform=android-23 --install-dir=/PATH/TO/STANDALONE/TOOLCHAINS --toolchain=aarch64-linux-androideabi-4.9 # Configure with cmake cmake -DCMAKE_TOOLCHAIN_FILE=../make/toolchain-android-arm64.cmake -DANDROID_TOOLCHAIN=/PATH/TO/STANDALONE/TOOLCHAINS -DDR_COPY_TO_DEVICE=OFF
please make commit to documentation as well
thanks
Files under ext/drsyms/libelftc-android-aarch64 is static library, for security and trust, I won't build and upload them.
Docs also need to be update after merge. My steps to build:
# Download ndk, any ndk with gcc should be okay. wget https://dl.google.com/android/repository/android-ndk-r14b-linux-x86_64.zip # Uncompress ... # Make standalone toolchains for android-23 android-ndk-r14b/build/tools/make-standalone-toolchain.sh --arch=arm64 --platform=android-23 --install-dir=/PATH/TO/STANDALONE/TOOLCHAINS --toolchain=aarch64-linux-androideabi-4.9 # Configure with cmake cmake -DCMAKE_TOOLCHAIN_FILE=../make/toolchain-android-arm64.cmake -DANDROID_TOOLCHAIN=/PATH/TO/STANDALONE/TOOLCHAINS -DDR_COPY_TO_DEVICE=OFF
The error below occurs in the same steps to build: make[2]: *** No rule to make target '../ext/drsyms/libelftc-android-aarch64/lib64/libdwarf.a', needed by 'ext/lib64/release/libdrsyms.so'. Stop.
Files under ext/drsyms/libelftc-android-aarch64 is static library, for security and trust, I won't build and upload them. Docs also need to be update after merge. My steps to build:
# Download ndk, any ndk with gcc should be okay. wget https://dl.google.com/android/repository/android-ndk-r14b-linux-x86_64.zip # Uncompress ... # Make standalone toolchains for android-23 android-ndk-r14b/build/tools/make-standalone-toolchain.sh --arch=arm64 --platform=android-23 --install-dir=/PATH/TO/STANDALONE/TOOLCHAINS --toolchain=aarch64-linux-androideabi-4.9 # Configure with cmake cmake -DCMAKE_TOOLCHAIN_FILE=../make/toolchain-android-arm64.cmake -DANDROID_TOOLCHAIN=/PATH/TO/STANDALONE/TOOLCHAINS -DDR_COPY_TO_DEVICE=OFFThe error below occurs in the same steps to build: make[2]: *** No rule to make target '../ext/drsyms/libelftc-android-aarch64/lib64/libdwarf.a', needed by 'ext/lib64/release/libdrsyms.so'. Stop.
We switched to libdw from elfutils in #5926 for Linux; probably it is best to do the same for Android, which would avoid the need for the static lib{dwarf,elf}.a -- though the demangler in libelftc.a is still in use as libdw does not provide demangling (but this is a much smaller library with much fewer changes over time).
Files under ext/drsyms/libelftc-android-aarch64 is static library, for security and trust, I won't build and upload them. Docs also need to be update after merge. My steps to build:
# Download ndk, any ndk with gcc should be okay. wget https://dl.google.com/android/repository/android-ndk-r14b-linux-x86_64.zip # Uncompress ... # Make standalone toolchains for android-23 android-ndk-r14b/build/tools/make-standalone-toolchain.sh --arch=arm64 --platform=android-23 --install-dir=/PATH/TO/STANDALONE/TOOLCHAINS --toolchain=aarch64-linux-androideabi-4.9 # Configure with cmake cmake -DCMAKE_TOOLCHAIN_FILE=../make/toolchain-android-arm64.cmake -DANDROID_TOOLCHAIN=/PATH/TO/STANDALONE/TOOLCHAINS -DDR_COPY_TO_DEVICE=OFFThe error below occurs in the same steps to build: make[2]: *** No rule to make target '../ext/drsyms/libelftc-android-aarch64/lib64/libdwarf.a', needed by 'ext/lib64/release/libdrsyms.so'. Stop.
Hi @summershrimp , I use the [summershrimp:android-aarch64] branch and the same build steps as yours mentioned above, but I encounter the error. It seems to be because the ext/drsyms/libelftc-android-aarch64/ directory does not contain any library, so I copy all the *.a libraries in libelftc-aarch64 directory to the libelftc-android-aarch64. However, a new error occurs:
.../android-ndk-r14b/_install/bin/../lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld: warning: libm.so, needed by ../lib64/release/libdrsyms.so, not found (try using -rpath or -rpath-link)
../lib64/release/libdrsyms.so: undefined reference to __ctype_b_loc' ../lib64/release/libdrsyms.so: undefined reference to __assert_fail'
../lib64/release/libdrsyms.so: undefined reference to __fxstat' ../lib64/release/libdrsyms.so: undefined reference to __errno_location'
collect2: error: ld returned 1 exit status
make[2]: *** [ext/drsyms/CMakeFiles/drsyms_bench.dir/build.make:99: ext/drsyms_bench] Error 1
make[1]: *** [CMakeFiles/Makefile2:1574: ext/drsyms/CMakeFiles/drsyms_bench.dir/all] Error 2
Could you please help provide some ideas to solve the problem? Thanks!
Files under ext/drsyms/libelftc-android-aarch64 is static library, for security and trust, I won't build and upload them. Docs also need to be update after merge. My steps to build:
# Download ndk, any ndk with gcc should be okay. wget https://dl.google.com/android/repository/android-ndk-r14b-linux-x86_64.zip # Uncompress ... # Make standalone toolchains for android-23 android-ndk-r14b/build/tools/make-standalone-toolchain.sh --arch=arm64 --platform=android-23 --install-dir=/PATH/TO/STANDALONE/TOOLCHAINS --toolchain=aarch64-linux-androideabi-4.9 # Configure with cmake cmake -DCMAKE_TOOLCHAIN_FILE=../make/toolchain-android-arm64.cmake -DANDROID_TOOLCHAIN=/PATH/TO/STANDALONE/TOOLCHAINS -DDR_COPY_TO_DEVICE=OFFThe error below occurs in the same steps to build: make[2]: *** No rule to make target '../ext/drsyms/libelftc-android-aarch64/lib64/libdwarf.a', needed by 'ext/lib64/release/libdrsyms.so'. Stop.
Hi @summershrimp , I use the [summershrimp:android-aarch64] branch and the same build steps as yours mentioned above, but I encounter the error. It seems to be because the ext/drsyms/libelftc-android-aarch64/ directory does not contain any library, so I copy all the *.a libraries in libelftc-aarch64 directory to the libelftc-android-aarch64. However, a new error occurs: .../android-ndk-r14b/_install/bin/../lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld: warning: libm.so, needed by ../lib64/release/libdrsyms.so, not found (try using -rpath or -rpath-link) ../lib64/release/libdrsyms.so: undefined reference to
__ctype_b_loc' ../lib64/release/libdrsyms.so: undefined reference to__assert_fail' ../lib64/release/libdrsyms.so: undefined reference to__fxstat' ../lib64/release/libdrsyms.so: undefined reference to__errno_location' collect2: error: ld returned 1 exit status make[2]: *** [ext/drsyms/CMakeFiles/drsyms_bench.dir/build.make:99: ext/drsyms_bench] Error 1 make[1]: *** [CMakeFiles/Makefile2:1574: ext/drsyms/CMakeFiles/drsyms_bench.dir/all] Error 2Could you please help provide some ideas to solve the problem? Thanks!
You can't use these library from libelftc-aarch64, because they are built against glibc. Please build your own libelftc-android-aarch64 with android-ndk
You can follow the steps here to build it https://github.com/DynamoRIO/dynamorio/blob/af9d1063fc2b4e97f808d429d2149cb8c06baf60/ext/drsyms/libelftc-android-arm/HOWTOBUILD but replace arm-xxx to aarch64-xxx (or maybe arm64-xxx). Just use standalone toolchains for android-23 mentioned above.
I didn't submit these prebuilt binary because that these binary would be better built and submit by dr's core team so other users can trust these binary not having backdoors or any unintended modification.
@summershrimp Thanks for replying! I build it successfully by your suggestions and copy the package to my aarch64-android14 platform. The output is as below when I run a test and it seems abnormal:
$ bin64/drrun -verbose -c samples/bin64/libbbsize.so -- ls INFO: default root: /data/DynamoRIO-Android-9.93.19479/bin64/.. INFO: default toolconfig dir: /data/DynamoRIO-Android-9.93.19479/bin64/../tools INFO: client 0 path: /data/DynamoRIO-Android-9.93.19479/samples/bin64/libbbsize.so INFO: targeting application: "/system/bin/toybox" INFO: app cmdline: "ls" INFO: configuration directory is "/data/local/tmp/.dynamorio" INFO: will exec /system/bin/toybox INFO: registering client with id=0 path=|/data/DynamoRIO-Android-9.93.19479/samples/bin64/libbbsize.so| ops=|| Trap (core dumped)
@summershrimp Thanks for replying! I build it successfully by your suggestions and copy the package to my aarch64-android14 platform. The output is as below when I run a test and it seems abnormal:
$ bin64/drrun -verbose -c samples/bin64/libbbsize.so -- ls INFO: default root: /data/DynamoRIO-Android-9.93.19479/bin64/.. INFO: default toolconfig dir: /data/DynamoRIO-Android-9.93.19479/bin64/../tools INFO: client 0 path: /data/DynamoRIO-Android-9.93.19479/samples/bin64/libbbsize.so INFO: targeting application: "/system/bin/toybox" INFO: app cmdline: "ls" INFO: configuration directory is "/data/local/tmp/.dynamorio" INFO: will exec /system/bin/toybox INFO: registering client with id=0 path=|/data/DynamoRIO-Android-9.93.19479/samples/bin64/libbbsize.so| ops=|| Trap (core dumped)
Have you applied the workaround patch mentioned at #3543 ? If you already applied it, I've no idea what happend. You need to debug it yourself. Maybe just start with running without any client? And also DR may use some unusual signal to work, these signal may catched by the android bionic libc?
@summershrimp @derekbruening AArch64-Android is not supported by DynamoRIO currently. Can I compile the project statically against AArch64-Linux and copy the bin files to AArch64-Android to run? If yes, how to compile it statically? Thanks!
@summershrimp @derekbruening AArch64-Android is not supported by DynamoRIO currently. Can I compile the project statically against AArch64-Linux and copy the bin files to AArch64-Android to run? If yes, how to compile it statically? Thanks!
No, Android is different. As you can see from the details in the filed issues, the main problem is private loading of client libraries on Android with its undocumented loader behavior. Linux glibc is moving in the direction of having similar problems: see the long-term solution dicsussion in the 3+ comments starting at https://github.com/DynamoRIO/dynamorio/issues/5437#issuecomment-1083108099. If you would like to work on a solution that would be appreciated by the community. If your client does not use 3rd-party libraries (only uses the DR API, which does provide quite a few features), you will bypass the private loader issues.
Since the newer NDKs only support LLVM-based toolchains (as noted in this discussion , this changelog r23 or r24 and my test), I am unable to build DynamoRIO using the latest NDKs (e.g., r26) because the GNU assembler (gas) has been removed in ndk-r24.
I want to use DynamoRIO on Android 14 AArch64 devices, benefiting from new features like memory tagging. Therefore, using the latest NDK is essential (my tests show that r27c works fine with memtag extension).
Here are my questions:
- Can DynamoRIO be built using only an LLVM-based toolchain? ~~I believe the answer is no, as indicated here~~ (It has been checked useless, see this). ~~If that is the case, which parts of DynamoRIO can only be compiled with GNU tools? Could you provide an example? (I am not very familiar with DynamoRIO's core source code.)~~ ~~I have now prepared it for compilation~~ (~~treating ld.lld as gold,~~ but it does not work...). There may still be some issues resulting from this approach.
- ~~I would like to modify the CMake configuration and some source files to enable support for an LLVM-based toolchain.~~ It seems some minor changes are okay. Do I need to take this approach to resolve the compilation issue on Android AArch64, or can I simply compile the clients using a newer NDK toolchain?
- If I want to submit a pull request to add an option in the source code for compiling with the LLVM toolchain, but currently I only have time to make changes to the DynamoRIO core, can I still submit a pull request?
- Previously, I built aarch64 android version using an older NDK version and this commit. However, it crashes with a "Segmentation Fault" error when I run
path/to/drrun -- ls. I tested helloworld.c (a simpleprintfprogram), and it works fine with the sample DynamoRIO clients. Could this problem be related to the private loader of Android Bionic?
@jimmypury it may be best to move discussion out of this PR into #2154. I replied at https://github.com/DynamoRIO/dynamorio/issues/2154#issuecomment-2494417883
PR #7101 now provides the ability to build for aarch64 Android using newer Clang-based NDKs. #7101 does not implement running on aarch64 Android, so this PR may still be of use until future patches allow DR to run on aarch64 Android if people have got it running from this patch in the past.
