Android cross-compilation: Build environment overrides ldc configuration?

[tmarplatt]

I'm not familiar enough with ldc compilation and can't resolve this problem on my own. I'm reaching out for help if that's allowed to do in a issue ticket.

I'm trying to set up ldc 1.19.0-beta2 cross-compilation under python-for-android, in order to be able to build Android packages that use D language code through pyd.

For each Python module that is being packaged, python-for-android sets up a contained build environment and follows scripted cross-compilation rules (called "recipes"). I'm writing one for pyd-enabled modules, which 1) fetches and installs ldc, 2) fetches the ldc prebuilt ARM runtime libraries, 3) extends etc/ldc2.conf, and 4) patches pyd to pass the correct -mtriple to ldc (pyd takes care of compiling the D module as a shared library, then copies it into the Python installation).

But I believe the build environment is interfering. E.g. consider ldmd2 having cross-compiled our pyd enabled D module into an ARM eabi object file temp.o. When pyd attempts to run the command: ldc2 -v -mtriple=armv7a--linux-androideabi -shared -d-debug -of build/lib.linux-x86_64-3.7/worldwrapper.cpython-37m.so build/temp.linux-x86_64-3.7/infra/temp.o it outputs:

version   1.19.0-beta2 (DMD v2.089.0, LLVM 9.0.0)
config    /home/tmarplatt/pyd/testapp/recipes/pyd-module/ldc/ldc-1.19.0-beta2/etc/ldc2.conf (armv7a-unknown-linux-android)
predefs   LDC all D_Version2 assert D_ModuleInfo D_Exceptions D_TypeInfo ARM ARM_SoftFP D_HardFloat LittleEndian D_PIC linux Posix Android CRuntime_Bionic LDC_LLVM_900
GC stats  0M used, 1M free, 1M total
/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang build/temp.linux-x86_64-3.7/infra/temp.o -shared -o build/lib.linux-x86_64-3.7/worldwrapper.cpython-37m.so -fuse-ld=bfd -v -L/home/tmarplatt/pyd/testapp/recipes/pyd-module/ldc/ldc-1.19.0-beta2/bin/../lib-armv7a-32 -L/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/android-build -lphobos2-ldc -ldruntime-ldc -lpython3.7m -Wl,--gc-sections -ldl -lm 
Android (5058415 based on r339409) clang version 8.0.2 (https://android.googlesource.com/toolchain/clang 40173bab62ec746213857d083c0e8b0abb568790) (https://android.googlesource.com/toolchain/llvm 7a6618d69e7e8111e1d49dc9e7813767c5ca756a) (based on LLVM 8.0.2svn)
Target: x86_64-unknown-linux-gnu

Since clang tries to target an x86_64 architecture despite of the specified -mtriple, linking fails. *

Here is the full environment set by python-for-android. Could any of these flags be the culprit?

CFLAGS = -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -I/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Include
CXXFLAGS = -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC
CPPFLAGS = -DANDROID -D__ANDROID_API__=21 -I/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/sysroot/usr/include/arm-linux-androideabi -I/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/python-installs/tmarpp/include/python3.7
LDFLAGS =   -L/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/libs_collections/tmarpp/armeabi-v7a -L/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/android-build -lpython3.7m
LDLIBS = -lm
CC = /home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC
CXX = /home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC
AR = arm-linux-androideabi-ar
RANLIB = arm-linux-androideabi-ranlib
STRIP = arm-linux-androideabi-strip --strip-unneeded
MAKE = make -j3
READELF = arm-linux-androideabi-readelf
NM = arm-linux-androideabi-nm
LD = arm-linux-androideabi-ld
ARCH = armeabi-v7a
NDK_API = android-21
TOOLCHAIN_PREFIX = arm-linux-androideabi
TOOLCHAIN_VERSION = 4.9
LDSHARED = /home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
BUILDLIB_PATH = /home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/hostpython3/desktop/hostpython3/native-build/build/lib.linux-x86_64-3.7
PATH = /home/tmarplatt/pyd/testapp/recipes/pyd-module/ldc/ldc-1.19.0-beta2/bin:/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin:/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin:/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86/bin/:/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/:/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b:/home/tmarplatt/.buildozer/android/platform/android-sdk/tools:/home/tmarplatt/.buildozer/android/platform/apache-ant-1.9.4/bin:/home/tmarplatt/pyd/bin:~/tldr:/home/tmarplatt/.local/bin:/home/tmarplatt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/tmarplatt/.local/bin/::/home/tmarplatt/bin/
PYTHONNOUSERSITE = 1
LANG = en_GB.UTF-8
PYTHONPATH = /home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib:/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages:/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/hostpython3/desktop/hostpython3/native-build/build/lib.linux-x86_64-3.7:/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/hostpython3/desktop/hostpython3/native-build/build/temp.linux-x86_64-3.7:/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/hostpython3/desktop/hostpython3/native-build/build/scripts-3.7
DC = ldc2
DMD = ldmd2
LIBRARY_PATH = /home/tmarplatt/pyd/testapp/recipes/pyd-module/ldc/ldc-1.19.0-beta2/lib
LD_LIBRARY_PATH = /home/tmarplatt/pyd/testapp/recipes/pyd-module/ldc/ldc-1.19.0-beta2/lib

Here's the etc/ldc2.conf extension:

"armv7a-.*-linux-android":
{
    switches = [
        "-defaultlib=phobos2-ldc,druntime-ldc,python3.7m",
        "-link-defaultlib-shared=false",
        "-linker=bfd",
        "-mcpu=cortex-a8",
        "-gcc=/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang",
        "-Xcc=-v",
        //"-Xcc=--target=armv7a-unknown-linux-android21",
    ];
    lib-dirs = [
        "%%ldcbinarypath%%/../lib-armv7a-32",
        "/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/android-build",
    ];
};

* If I try to work around this issue, e.g. by passing the correct --target to the linker (via etc/ldc2.conf), it links "successfully" but then the shared library is unusable: ImportError: dlopen failed: cannot locate symbol "_tlsstart" referenced by "/data/data/org.tmarplatt.testapp.tmarpp/files/app/_python_bundle/site-packages/worldwrapper.so"... And I shouldn't have to do that to begin with, as ldc2 is natively capable of passing the right linker parameters.

[kinke]

I guess the simple fix is to use a pre-configured linker driver, i.e., -gcc=.../bin/armv7a-linux-androideabi21-clang instead of the native -gcc=.../bin/clang. That's how it's handled in the Azure Pipelines script, and suggested in the Wiki.

2. fetches the ldc prebuilt ARM runtime libraries, 3) extends etc/ldc2.conf

With #3244 and hopefully 1.19 final, we'll also provide a native Android/AArch64 package, and extend both armv7a/aarch64 packages by prebuilt i686/x86_64 druntime/Phobos libraries. So there'll be prebuilt libs for all 4 Android platforms.

These steps, fetching a prebuilt LDC package for a cross-compilation target, extracting the libs and extending ldc2.conf, would come in handy as a generic little tool, supporting all prebuilt packages/targets. - Thx for the integration work. :)

[kinke]

Wrt. _tls{start,end}, those symbols are generated for Android by LDC in the object file containing D main(). Joakim mentioned this Android-specific requirement for shared libs in the Wiki; it's related to custom TLS emulation.

[tmarplatt]

I guess the simple fix is to use a pre-configured linker driver, i.e., -gcc=.../bin/armv7a-linux-androideabi21-clang instead of the native -gcc=.../bin/clang. That's how it's handled in the Azure Pipelines script, and suggested in the Wiki.

That was a simple fix indeed, thanks.

From the Wiki:

Running multiple D shared libraries is currently unsupported on Android, only a single D shared library that statically links against the D runtime will work.

Is ldc already linking statically against the D runtime libs when passed -shared -mtriple=armv7a--linux-androideabi? Or do I have to that myself...

[kinke]

Is ldc already linking statically

There's no triple-specific hardcoded logic regarding this, i.e., this has to be taken care of via a -link-defaultlib-shared=false default switch for Android targets in ldc2.conf.

[kinke]

Btw, IIRC, the D object file containing a dummy main() should be the first D object file in the linker cmdline, so that _tls{start,end} cover the proper, full range.

[tmarplatt]

Well now the linker outputs a number of "R_ARM_TLS_LDO32 used with non-TLS symbol" warnings:

[DEBUG]:   	 "/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.bfd" -X --enable-new-dtags --eh-frame-hdr -m armelf_linux_eabi -shared -o build/lib.linux-x86_64-3.7/worldwrapper.cpython-37m.so /home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/21/crtbegin_so.o -L/home/tmarplatt/pyd/testapp/recipes/pyd-module/ldc/ldc-1.19.0-beta2/bin/../lib-armv7a-32 -L/home/tmarplatt/pyd/testapp/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/android-build -L/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/8.0.2/lib/linux/arm -L/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/armv7-a -L/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/../lib/armv7-a -L/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/21 -L/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi -L/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib/../lib -L/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/../../lib -L/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/armv7-a -L/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib build/temp.linux-x86_64-3.7/infra/temp.o -lphobos2-ldc -ldruntime-ldc -lpython3.7m --gc-sections -ldl -lm -lgcc -ldl -lc -lgcc -ldl /home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/21/crtend_so.o
[DEBUG]:   	/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.bfd: build/temp.linux-x86_64-3.7/infra/temp.o(.debug_info+0x8a): R_ARM_TLS_LDO32 used with non-TLS symbol PyDateTimeAPI
[DEBUG]:   	/home/tmarplatt/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.bfd: build/temp.linux-x86_64-3.7/infra/temp.o(.debug_info+0x9042): R_ARM_TLS_LDO32 used with non-TLS symbol _D3pyd6thread10isAttachedb
...

And dozens more, all referring to pyd symbols. Linking doesn't fail but then the library won't load. (I believe it's segfaulting but I haven't yet figured out how to get a stack trace from the phone—logcat show nothing.)

[kinke]

These warnings are due to our modded LLVM not taking care of debuginfos for TLS variables; not specifying -g gets rid of them.

[kinke]

Please retry with the latest beta, the segfault may likely be fixed now.

[tmarplatt]

Thanks I will retry as soon as I can, since I've moved my development setup to a different machine and have yet to settle down.