[question] Cross-compiling with arm toolchain and unit tests

[MitchellBot]

What is your question?

My goal is to not only cross compile from Linux x86_64 (build) to Linux Armv8 x64 (host) (which is working fine) but to also run my unit tests as part of the conan create command, but I'm getting an error from qemu-aarch64 during test discovery.

Conan version 2.0.14

I'm currently using the example Arm Toolchain https://docs.conan.io/2/examples/cross_build/toolchain_packages.html with only one very minor fix ({valid_archs.join(',')} throws an error but removing the .join(',') fixes).

The project itself is CMake where GTest-based unit tests are in a 'Tests' directory with its own CMakeLists.txt and runs the discovery command gtest_discover_tests.

The error is a missing file ld-linux-aarch64.so.1 which exists in: ~/.conan2/p/b/armtoa3ace3b2f5614/p/aarch64-none-linux-gnu/libc/lib. I believe I just need to set a variable or copy this file into where it's expected, but I'm not sure how to accomplish this.

error:

qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory
LD_LIBRARY_PATH=+/home/user/.conan2/p/b/armtoa3ace3b2f5614/p/lib:
CMake Error at /usr/share/cmake/Modules/GoogleTestAddTests.cmake:78 (message):
  Error running test executable.

    Path: '/home/user/.conan2/p/b/cpage926c22ab7b67b/b/build/Debug/test/Program_Tests'
    Result: 255
    Output:
      

Call Stack (most recent call first):
  /usr/share/cmake/Modules/GoogleTestAddTests.cmake:174 (gtest_discover_tests_impl)


gmake[2]: *** [Tests/CMakeFiles/Program_Tests.dir/build.make:253: test/Program_Tests] Error 1
gmake[2]: *** Deleting file 'test/Program_Tests'
gmake[1]: *** [CMakeFiles/Makefile2:945: Tests/CMakeFiles/Program_Tests.dir/all] Error 2
gmake: *** [Makefile:101: all] Error 2

linux-armv8.yml:

[settings]
os=Linux
arch=armv8
compiler=gcc
compiler.cppstd=gnu14
compiler.libcxx=libstdc++11
compiler.version=13

[options]
boost/*:shared=True
boost/*:without_test=True
msgpack-cxx/*:use_boost=False

[tool_requires]
armtoolchain/13.2

Toolchains/arm/conanfile.py:

import os
from conan import ConanFile
from conan.tools.files import get, copy, download
from conan.errors import ConanInvalidConfiguration
from conan.tools.scm import Version

class ArmToolchainPackage(ConanFile):
    name = "armtoolchain"
    version = "13.2"

    license = "GPL-3.0-only"
    homepage = "https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads"
    description = "Conan package for the ARM toolchain, targeting different Linux ARM architectures."
    settings = "os", "arch"
    package_type = "application"

    def _archs32(self):
        return ["armv6", "armv7", "armv7hf"]
    
    def _archs64(self):
        return ["armv8", "armv8.3"]

    def _get_toolchain(self, target_arch):
        if target_arch in self._archs32():
            return ("arm-none-linux-gnueabihf", 
                    "df0f4927a67d1fd366ff81e40bd8c385a9324fbdde60437a512d106215f257b3")
        else:
            return ("aarch64-none-linux-gnu", 
                    "12fcdf13a7430655229b20438a49e8566e26551ba08759922cdaf4695b0d4e23")

    def validate(self):
        if self.settings.arch != "x86_64" or self.settings.os != "Linux":
            raise ConanInvalidConfiguration(f"This toolchain is not compatible with {self.settings.os}-{self.settings.arch}. "
                                            "It can only run on Linux-x86_64.")

        valid_archs = self._archs32() + self._archs64()
        if self.settings_target.os != "Linux" or self.settings_target.arch not in valid_archs:
            raise ConanInvalidConfiguration(f"This toolchain only supports building for Linux-{valid_archs}. "
                                           f"{self.settings_target.os}-{self.settings_target.arch} is not supported.")

        if self.settings_target.compiler != "gcc":
            raise ConanInvalidConfiguration(f"The compiler is set to '{self.settings_target.compiler}', but this "
                                            "toolchain only supports building with gcc.")

        if Version(self.settings_target.compiler.version) >= Version("14") or Version(self.settings_target.compiler.version) < Version("13"):
            raise ConanInvalidConfiguration(f"Invalid gcc version '{self.settings_target.compiler.version}'. "
                                            "Only 13.X versions are supported for the compiler.")

    def source(self):
        download(self, "https://developer.arm.com/GetEula?Id=37988a7c-c40e-4b78-9fd1-62c20b507aa8", "LICENSE", verify=False)

    def build(self):
        toolchain, sha = self._get_toolchain(self.settings_target.arch)
        get(self, f"https://developer.arm.com/-/media/Files/downloads/gnu/13.2.rel1/binrel/arm-gnu-toolchain-13.2.rel1-x86_64-{toolchain}.tar.xz",
            sha256=sha, strip_root=True)            

    def package(self):
        toolchain, _ = self._get_toolchain(self.settings_target.arch)
        dirs_to_copy = [toolchain, "bin", "include", "lib", "libexec"]
        for dir_name in dirs_to_copy:
            copy(self, pattern=f"{dir_name}/*", src=self.build_folder, dst=self.package_folder, keep_path=True)
        copy(self, "LICENSE", src=self.build_folder, dst=os.path.join(self.package_folder, "licenses"), keep_path=False)

    def package_id(self):
        self.info.settings_target = self.settings_target
        # We only want the ``arch`` setting
        self.info.settings_target.rm_safe("os")
        self.info.settings_target.rm_safe("compiler")
        self.info.settings_target.rm_safe("build_type")

    def package_info(self):
        toolchain, _ = self._get_toolchain(self.settings_target.arch)
        self.cpp_info.bindirs.append(os.path.join(self.package_folder, toolchain, "bin"))

        self.conf_info.define("tools.build:compiler_executables", {
            "c":   f"{toolchain}-gcc",
            "cpp": f"{toolchain}-g++",
            "asm": f"{toolchain}-as"
        })

Can anyone tell me what needs to be set and where? qemu-aarch64 appears to be looking for an absolute path ('/lib/ld-linux-aarch64.so.1'), can this be corrected in a variable? Since LD_LIBRARY_PATH is already set (as the error shows), I'm assuming that it's not that but something else.

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[MitchellBot]

Some manual hacks got it working but now I need to figure out how to get those hacks into the toolchain recipe itself. After much trial and error, the following commands resolved all my issues:

  sudo ln -f -s /home/user/.conan2/p/b/armtoa3ace3b2f5614/p/aarch64-none-linux-gnu/libc/lib/ld-linux-aarch64.so.1 /lib

  ln -f -s /home/user/.conan2/p/b/armtoa3ace3b2f5614/p/aarch64-none-linux-gnu/lib64/libstdc++.so.6 /home/user/.conan2/p/b/armtoa3ace3b2f5614/p/lib
  ln -f -s /home/user/.conan2/p/b/armtoa3ace3b2f5614/p/aarch64-none-linux-gnu/libc/lib64/libm.so.6 /home/user/.conan2/p/b/armtoa3ace3b2f5614/p/lib
  ln -f -s /home/user/.conan2/p/b/armtoa3ace3b2f5614/p/aarch64-none-linux-gnu/lib64/libgcc_s.so.1 /home/user/.conan2/p/b/armtoa3ace3b2f5614/p/lib
  ln -f -s /home/user/.conan2/p/b/armtoa3ace3b2f5614/p/aarch64-none-linux-gnu/libc/lib64/libc.so.6 /home/user/.conan2/p/b/armtoa3ace3b2f5614/p/lib

The worrisome one is the first sudo command. Since qemu-aarch64 was looking in my own /lib path, I had to create a symbolic link in my own /lib to point to the armtoolchain conan package.

The rest was because LD_LIBRARY_PATH is set to a location inside the toolchain's directory in Conan's cache, but didn't include some of the required libs from elsewhere.

Obviously none of this is sustainable since it will all break when I download a new version of the toolchain. So the big question remains, how do I get qemu-aarch64 to look in the right place and how can I create these links from within the conan recipe?

[memsharded]

Hi @MitchellBot

Thanks for your report.

A couple of quick things: First, please upgrade and try with Conan 2.2. I don't think there will be a big difference, but 2.0.14 is already 5 months old, and development this time has been wild, many improvements and fixes, so worth trying.

Then, what Linux distro are you using? We are mostly using Ubuntu, and the example has CI and is tested. Maybe there is some difference in the distro? Probably not, but just in case. It looks more like the example in the docs is "basic" and might not be covering more advanced cases.

If you can share a simple consumer project like the one with tests that you are commenting that we could reproduce, that would also help. Thanks!

[MitchellBot]

@memsharded

Oracle Linux 8.9 on WSL.

I updated to Conan 2.2 with pip3 install conan --update, removed my symbolic link, and removed my toolchain recipe (to undo the other symbolic links) and I'm back to the same error.

qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory
LD_LIBRARY_PATH=+/home/user/.conan2/p/armtoa08836ba21783/p/lib:

I'll try to make a simple project that can repro this but it may take some time.

[MitchellBot]

@memsharded I stole quite a bit from examples to create this: https://github.com/MitchellBot/conantest

Build the armtoolchain and then from the sum directory run conan create . -pr:b ../profiles/linux-x86_64-Debug.yml -pr:h ../profiles/linux-armv8-Debug.yml The project builds and tests fine for me when not setting a profile, or setting both to x86_64. But I get the same error as originally reported if I run with the armv8 host profile.

[fschoenm]

@MitchellBot Have you tried setting CMAKE_BUILD_RPATH to your sysroot (e.g. in a CMake toolchain file)? I think CMake does not usually add the sysroot to the RPATH as it is part of the default library search path but this doesn't apply before your binary has been deployed.

[MitchellBot]

@fschoenm Wouldn't I want it set in the toolchain inside Conan's cache instead of my sysroot? The point is that qemu-aarch64 should be looking in the ~/.conan2 "armtoolchain" package directory for the ld-linux-aarch64.so.1 file. But I'm not sure how to set that up without manually creating symbolic links. It's also only the Test executable that needs it set.

[fschoenm]

@MitchellBot That is correct (the sysroot I was refering to is part of the compiler/toolchain package). In our cross-compiler package, we have a toolchain.cmake file in the root of the package_folder with the following settings:

set(CMAKE_BUILD_RPATH "${CMAKE_CURRENT_LIST_DIR}/{{ target_tuple }}/sysroot/lib")

When installing your project, CMake should remove the RPATH to make it work on the target system.

For qemu in particular you can also set a library path, in case the RPATH doesn't work:

set(CMAKE_CROSSCOMPILING_EMULATOR "/usr/bin/qemu-{{ target_arch }};-L;${CMAKE_CURRENT_LIST_DIR}/{{ target_tuple }}/sysroot")

[MitchellBot]

@fschoenm So I don't actually have qemu installed, nor can I find anything that provides ld-linux-aarch64.so.1 in yum or a qemu-system-aarch64 or qemu-aarch64 package for Oracle. But it doesn't seem to be necessary to have that installed since I can get everything to work with some hacky symbolic links. Everything I need seems to be on the system somewhere right now, I just can't get it to look in the correct places.

Doing an strace of CMake during the build process shows that qemu-aarch64 isn't being called as an executable at all (and is nowhere on the system anyway). Instead, I get the following:

[pid 39830] execve("/home/user/.conan2/p/b/sumbdfe24d154ef1/b/build/Debug/test/test_sum", ["/home/user/.conan2/p/b/sumb"..., "--gtest_list_tests"], 0x7ffc91929258 /* 69 vars */ <unfinished ...>
...
[pid 39830] access("/usr/gnemul/qemu-aarch64/lib/ld-linux-aarch64.so.1", F_OK) = -1 ENOENT (No such file or directory)
[pid 39830] open("/lib/ld-linux-aarch64.so.1", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)

So the test executable itself is what's being run and it only seems to be looking in two locations for the toolchain library.

I tried to modify your command and instead of running gtest_discover_tests(${TEST_PROJECT}) I changed it to add_test(NAME ${TEST_PROJECT} COMMAND $<TARGET_FILE:${TEST_PROJECT}> -L "${CMAKE_CURRENT_LIST_DIR}/{{ target_tuple }}/sysroot/lib") and that failed but now the qemu-aarch64 error is seen in the LastTest.log file:

Start testing: Apr 15 08:41 PDT
----------------------------------------------------------
1/1 Testing: test_sum
1/1 Test: test_sum
Command: "/home/user/.conan2/p/b/sum524e7275ab021/b/build/Debug/test/test_sum" "-L" "/home/user/.conan2/p/b/sum524e7275ab021/b/test/{{ target_tuple }}/sysroot/lib"
Directory: /home/user/.conan2/p/b/sum524e7275ab021/b/build/Debug/test
"test_sum" start time: Apr 15 08:41 PDT
Output:
----------------------------------------------------------
qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory
<end of output>
Test time =   0.00 sec
----------------------------------------------------------
Test Failed.
"test_sum" end time: Apr 15 08:41 PDT
"test_sum" time elapsed: 00:00:00
----------------------------------------------------------

End testing: Apr 15 08:41 PDT

Additionally, setting set(CMAKE_CROSSCOMPILING_EMULATOR "qemu-{{ target_arch }};-L;${CMAKE_CURRENT_LIST_DIR}/{{ target_tuple }}/sysroot") changes the error to:

[100%] Linking CXX executable test_sum
LD_LIBRARY_PATH=+/home/user/.conan2/p/armtoa08836ba21783/p/lib:
CMake Error at /usr/share/cmake/Modules/GoogleTestAddTests.cmake:78 (message):
  Error running test executable.

    Path: '/home/user/.conan2/p/b/sum7f39039eb1aba/b/build/Debug/test/test_sum'
    Result: No such file or directory
    Output:

In other words, the test executable is somehow disappearing (or being built in another path).

Doing set(CMAKE_BUILD_RPATH "${CMAKE_CURRENT_LIST_DIR}/{{ target_tuple }}/sysroot/lib") seemed to make no difference at all.

Also, running file on test_sum reveals the following:

ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

Maybe this interpreter value can be changed?

[MitchellBot]

Ok, I think I have it.

In my test/CMakeLists.txt I conditionally look for whether it's an aarch64 build and then I change the --dynamic-linker of just the test executable to the path of CMAKE_CXX_COMPILER_SYSROOT/../lib/ld-linux-aarch64.so.1:

if (${CMAKE_SYSTEM_PROCESSOR} STREQUAL "aarch64")
    set_target_properties(${TEST_PROJECT} PROPERTIES LINK_FLAGS 
        "-Wl,--dynamic-linker=${CMAKE_CXX_COMPILER_SYSROOT}/../lib/ld-linux-aarch64.so.1"
    )
endif ()

Then, for the other missing libs, I updated the toolchain/conanfile.py to add the other libdirs:

    def package_info(self):
        toolchain, _ = self._get_toolchain(self.settings_target.arch)
        self.cpp_info.bindirs.append(os.path.join(self.package_folder, toolchain, "bin"))
        self.cpp_info.libdirs.append(os.path.join(self.package_folder, toolchain, "libc", "lib64"))
        self.cpp_info.libdirs.append(os.path.join(self.package_folder, toolchain, "lib64"))

        self.conf_info.define("tools.build:compiler_executables", {
            "c":   f"{toolchain}-gcc",
            "cpp": f"{toolchain}-g++",
            "asm": f"{toolchain}-as"
        })

Build and test success.

I also verified the executable is ARM64 and has the proper dynamic linker:

file /home/user/.conan2/p/b/sum57526f2f58c6d/b/build/Debug/test/test_sum /home/user/.conan2/p/b/sum57526f2f58c6d/b/build/Debug/test/test_sum: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /home/user/.conan2/p/b/armto958de8f8a2f96/p/bin/../aarch64-none-linux-gnu/libc/usr/../lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

One other problem I had while trying to come up with this solution was that I couldn't find any way to add custom variables to the conan_toolchain.cmake (hence the relative path off of CMAKE_CXX_COMPILER_SYSROOT). I tried a few things in package_info and even created a generate function to use CMakeToolchain but no matter what I added, nothing showed up in any of the generators/* files.

If anyone has suggestions for improvements, or a better way (correct way?) to do this, or knows how I can add variables to the toolchain (https://github.com/MitchellBot/conantest/blob/main/toolchain/conanfile.py) that can be consumed in other packages, I'd love to hear it, but I think this is at least a partial success.

[memsharded]

One other problem I had while trying to come up with this solution was that I couldn't find any way to add custom variables to the conan_toolchain.cmake (hence the relative path off of CMAKE_CXX_COMPILER_SYSROOT).

Sorry we are struggling to find time to follow up this thread.

For this specific issue, you can use the conf tools.cmake.cmaketoolchain:user_toolchain, which is a list of files, that you can append, etc, your own custom toolchain .cmake files with the variables you need. Maybe this link helps: https://docs.conan.io/2/examples/tools/cmake/cmake_toolchain/inject_cmake_variables.html. (toolchain .cmake files can also be injected via conf_info of the tool_requires)

[proceduralnoisy]

Related issues: #16686 , #12782.

In the case of cross-compiling, I believe you should use the DISCOVERY_MODE option:

gtest_discover_tests(<target name> DISCOVERY_MODE PRE_TEST)

Cross-compilation is the use case given in the docs for the PRE_TEST mode:

In certain scenarios, like cross-compiling, this POST_BUILD behavior is not desirable. By contrast, PRE_TEST delays test discovery until just prior to test execution. This way test discovery occurs in the target environment where the test has a better chance at finding appropriate runtime dependencies.

[MitchellBot]

Related issues: #16686 , #12782.

In the case of cross-compiling, I believe you should use the DISCOVERY_MODE option:

gtest_discover_tests(<target name> DISCOVERY_MODE PRE_TEST)

Cross-compilation is the use case given in the docs for the PRE_TEST mode:

In certain scenarios, like cross-compiling, this POST_BUILD behavior is not desirable. By contrast, PRE_TEST delays test discovery until just prior to test execution. This way test discovery occurs in the target environment where the test has a better chance at finding appropriate runtime dependencies.

This had no effect, I still need to programmatically change the dynamic-linker of the aarch64 test executable before running it.