Cross compiling with CMake for Android

[keith]

Currently when building a project for Android, the rules_foreign_cc build seems to still think the build is happening for macOS. This results in downstream CMake config not being correctly set if they are changing behavior based on the fact that they are cross compiling, or if some CMake variables like ANDROID are set.

The easiest way to reproduce this is to add this:

if (NOT ANDROID)
  message(FATAL_ERROR "Expected ANDROID to be set, system name is: ${CMAKE_SYSTEM_NAME}")
endif()

at the bottom of cmake_hello_world_lib/static/src/CMakeLists.txt and then run bazel build //cmake_android:app. At this point if you cd into the BUILD_TMPDIR you can also see that CMake run check commands like this one:

/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang   \
 -D__ANDROID_API__=28 \
 -isystem/private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sysroot/usr/include/arm-linux-androideabi \
 -target armv7-none-linux-androideabi \
 -march=armv7-a \
 -mfloat-abi=softfp \
 -mfpu=vfpv3-d16 \
 -gcc-toolchain /private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64 \
 -fpic \
 -no-canonical-prefixes \
 -Wno-invalid-command-line-argument \
 -Wno-unused-command-line-argument \
 -funwind-tables \
 -fstack-protector-strong \
 -fno-addrsig \
 -Werror=return-type \
 -Werror=int-to-pointer-cast \
 -Werror=pointer-to-int-cast \
 -Werror=implicit-function-declaration  \
 --sysroot=/private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/platforms/android-28/arch-arm \
 -isystem /private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sources/cxx-stl/llvm-libc++/include \
 -isystem /private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sources/cxx-stl/llvm-libc++abi/include \
 -isystem /private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sources/android/support/include \
 -isystem/private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sysroot/usr/include  \
 -isysroot /Applications/Xcode-11.0.0b3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk \
 -mmacosx-version-min=10.14   \
 -o CMakeFiles/cmTC_8f5fe.dir/testCCompiler.c.o   \
 -c /var/folders/q5/7_wh20z93c74nj3tllwjl5nr0000gn/T/tmp.UDH7XLoL/CMakeFiles/CMakeTmp/testCCompiler.c

Where both --sysroot and -isysroot are passed, and the -isysroot value is the macOS SDK, which makes me think that CMake definitely doesn't understand that it should be compiling for Android.

It seems like the way rules_foreign_cc sets everything up doesn't handle cross compilation, is that something we want to handle in CMake specific ways?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_foreign_cc!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

[cpsauer]

@keith, could I ask you how this one ended up? I assume cross compiling for android is still broken like this? (If so, and we know a collaborator, perhaps we should ask to reopen? Looks like there are a fair number of issues around cross compiling.)

[cpsauer]

Thanks for reopening, @UebelAndre! I should take that to mean that this issue is still outstanding, yeah?

[UebelAndre]

I currently understand it to be. In general anything closed by the bot just means the issues has become stale and it's closed until someone shows interest in it again.

[keith]

I don't recall at this point. We definitely use rules_foreign_cc in envoy targeting android, so it works at least in some cases

[cpsauer]

Thanks for your reply, Keith. Hope you had a great 4th weekend!

[cpsauer]

Can confirm, unfortunately, that this issue still reproduces, both with ANDROID and isysroot'ing the macOS SDK.

[cpsauer]

Though you can get around the former with, for example

cache_entries = select({
        "@platforms//os:android" : {"ANDROID": "1"}
    }),

(If you have platform_mappings set up)

[cpsauer]

But sadly, that macOS SDK isysroot still torpedos the build, causing redefinition errors.

[cpsauer]

Aha! Okay, so you can get considerably further by setting the following instead.

            "CMAKE_SYSTEM_NAME": "Android",
            "CMAKE_CXX_COMPILER_WORKS": "1",
            "BUILD_SHARED_LIBS": "off",

But there are still NEON and STL issues further down the line in the example I was working with.

Seems like a considerable number of additional settings would have to be propagated from Bazel -> Cmake to make this work well out of the box.

[TimoPtr]

I had the same issue in my project. I found out that the foreign rule is not setting the CMAKE_SYSTEM_NAME anywhere so it means that it takes the host value even when you crossbuild.

My build was working on Linux but not on MacOS with the same error. I printed the value of CMAKE_SYSTEM_NAME in Linux and it was set by default to Linux, but on MacOS it was darwin.

Setting the CMAKE_SYSTEM_NAME to Android leads to other issue about the STL like @cpsauer was mentioning that's why CMAKE_CXX_COMPILER_WORKS was needed. But some libraries use this feature to detect flags so it cannot be disable.

Since the foreign rule is properly setting the crosstool toolchain the values and it works on Linux, I did set the CMAKE_SYSTEM_NAME to Linux even on MacOS and it works since the crosstool path are correct.

    "ANDROID": "ON", 
    "CMAKE_SYSTEM_NAME": "Linux",

[keith]

Interestingly I have a test over here with the new NDK infra https://github.com/bazelbuild/rules_android_ndk/pull/24 that fails with this same issue, but the same test works fine with the bazel core NDK infra 🤔

[keith]

Ok I at least discovered 1 interesting thing about this while debugging my issue above. When cmake checks if the compiler is valid today, it defaults to macOS but that silently succeeds anyways depending on what features are being used. For me this didn't come up in some cases today because -search_paths_first is what causes the failure (at least in some cases), and that flag is silently ignored (or maybe interpreted as something else?!?) by ld.gold, but not ignored by lld, which is why the new NDKs fail immediately.

[keith]

I wonder if it would be reasonable to make rules_foreign_cc set the 2 variables mentioned above:

    "ANDROID": "ON", 
    "CMAKE_SYSTEM_NAME": "Linux",

whenever we're building for android? Are there any known downsides to that as a workaround? It's a bit annoying to set as a user if you target more than just android

[jsharpe]

I wonder if it would be reasonable to make rules_foreign_cc set the 2 variables mentioned above:

    "ANDROID": "ON", 
    "CMAKE_SYSTEM_NAME": "Linux",

whenever we're building for android? Are there any known downsides to that as a workaround? It's a bit annoying to set as a user if you target more than just android

Seems reasonable but I'm not up to speed enough on the whole platform eco-system code, especially with the special cases of Android, to know how to do detect this within the cmake framework. More generally the CMake rules need some logic for cross compilation support in general; I suspect that to do this generically the user may need to provide platform -> CMAKE_SYSTEM_NAME mappings manually as I don't think there is a way to know a-priori for an arbritary platform what the underlying system actually is if an OS based constraint isn't provided (which AFAIK isn't mandatory?)

[keith]

For discussion: https://github.com/bazelbuild/rules_foreign_cc/pull/997

[keith]

hrm actually with my tests with envoy-mobile https://github.com/envoyproxy/envoy/tree/main/mobile + rules_android_ndk + my patch, my patch isn't enough. maybe there are some other variables we need to set for this?

[keith]

hrm i might have just been unlucky with zlib and something about its cmake config, because some others do appear to work fine 🤔

[cpsauer]

(Perhaps they condition less on platform or something?)

Keith, if zlib for mobile is the primary issue, might it make sense to just link against the SDK-provided zlibs? That is -lz to linkopts. That's what we do for rules boost and for curl over in https://github.com/hedronvision/bazel-make-cc-https-easy