cryptography icon indicating copy to clipboard operation
cryptography copied to clipboard

Published 20 hours ago • verified publisher icon pyca

Error when cross-compiling

Open th0ma7 opened this issue 3 years ago • 4 comments
trafficstars

Working on adapting the spksrc framework to allow cross-compiling python wheels wich needs rustc compiler. Framework currently works really well at building all kinds of cross-compiled or pure-python wheels. Currently adding the necessary bits so rust support makes it through when cross-compiling wheels with pip.

As such I'm expecting that there may be loose-ends on my current experimental code to get this going. My hope is that you may provide me with some pointers.

Currently the only platform where it works is when the target = host (e.g. x86_64)

Versions:

  • arch: armv5, armv7, armv7l, armv8 (aarch64), i686
  • python: 3.10.6
  • libffi: 3.4.2
  • cryptography: Any with rust support but focus on latest 38.0.1
  • cffi: 1.15.1 (from crossenv)
  • pip: 22.2.2
  • setuptools: 63.4.3
  • setuptools-rust: 1.5.1

Error hit on all targets:

      error: failed to run custom build command for `parking_lot_core v0.8.5`

      Caused by:
        process didn't exit successfully: `/tmp/pip-wheel-dd2cglmv/cryptography_97728d383bc6460da3ae3795423f15e8/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build-script-build` (exit status: 2)
        --- stderr
        /tmp/pip-wheel-dd2cglmv/cryptography_97728d383bc6460da3ae3795423f15e8/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build-script-build: 1: /tmp/pip-wheel-dd2cglmv/cryptography_97728d383bc6460da3ae3795423f15e8/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build-script-build: Syntax error: word unexpected (expecting ")")
      warning: build failed, waiting for other jobs to finish...

The Syntax error: word unexpected (expecting ")") looks particulary of interest.

When disabling clean-up of the wheel build directory on armv7 for instance, all files looked properly linked:

spksrc@spksrc:/tmp/pip-wheel-vw0xe0c0/cryptography_22355c16355d45bfbdd30c60ed6b3cc6/src/rust/target/release/build$ file parking_lot_core-50*/*
parking_lot_core-506a93a9972231b0/build-script-build:                    ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, with debug_info, not stripped
parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0:   ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, with debug_info, not stripped
parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.d: ASCII text
parking_lot_core-50aea4c2af63d9de/invoked.timestamp:                     ASCII text, with no line terminators
parking_lot_core-50aea4c2af63d9de/out:                                   directory

PR with on-going experimental changes to the framework: https://github.com/SynoCommunity/spksrc/pull/5435

NOTE: PR mentioned above isn't yet fully in sync with my local build. Use logs below in the meantime.

build-aarch64-7.0.log build-armv7-7.0.log build-x64-7.0.log

th0ma7 avatar Sep 20 '22 00:09 th0ma7

Well, that's a fascinating error.

Looking at build-aarch64-7.0.log, is this supposed to be a cross-compilation from an x86-64 host?

alex avatar Sep 20 '22 02:09 alex

Well, that's a fascinating error.

Looking at build-aarch64-7.0.log, is this supposed to be a cross-compilation from an x86-64 host?

Yes indeed it is... with a bit of black magic perhaps :shrug:

th0ma7 avatar Sep 20 '22 02:09 th0ma7

Looks like a lot of black magic :-) A quick look at the log suggests that even host binaries are being built with a --target of aarch64. Host binaries, are, well, supposed to run on the host, meaning they should be x86. So however cross-compilation is being specified, it's somehow being overzealous

alex avatar Sep 20 '22 02:09 alex

That explains a lot already, thnx. In this specific case I was using the following wich does indeed enforce --target:

CARGO_HOME="/usr/local"
CARGO="/usr/local/bin/cargo"
RUSTC="/usr/local/bin/rustc"
RUSTFLAGS="--target=aarch64-unknown-linux-gnu -Clinker=/home/spksrc/rustc/spksrc/toolchain/syno-aarch64-7.0/
work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc -Car=/home/spksrc/rustc/spksrc/toolchain/syno-aarch64
-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-ar"

My other attempt (of many) was such as:

CARGO_HOME="/usr/local"
CARGO="/usr/local/bin/cargo"
RUSTC="/usr/local/bin/rustc"
CARGO_TARGET="armv7-unknown-linux-gnueabihf"
CARGO_TARGET_ARMV7_UNKNOWN_LINUX_GNUEABIHF_AR="/home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-u
nknown-linux-gnueabi/bin/arm-unknown-linux-gnueabi-ar"
CARGO_TARGET_ARMV7_UNKNOWN_LINUX_GNUEABIHF_LINKER="/home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/a
rm-unknown-linux-gnueabi/bin/arm-unknown-linux-gnueabi-gcc"

And failed such as the following log output: build-armv7-7.0.log

I believe when compiling "host" tools it takes into consideration the entire environment pointing towards the cross-compiling toolchain:

        = note: "cc" "-m64" "/tmp/rustcrongG1/symbols.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.0.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.1.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.10.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.11.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.12.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.13.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.14.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.15.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.2.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.3.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.4.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.5.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.6.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.7.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.8.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.build_script_build.b8003d9b-cgu.9.rcgu.o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0.3sb9v7w1vq5gqsq8.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/deps" "-L" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-8f1929c73c3f8167.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-e359d865975ccf21.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-b886fd10c5a7c7c0.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-9d7c322d48daa475.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-08ae1606a951cabe.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-682a81c4b2133b72.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-a73b3512c88de071.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-7b5ec4c918d9f957.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-65c63cf3af0af657.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-4a53f0a2785abc6a.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-868e2d515c28d027.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-394ad2d73aede76a.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-19c77e4dc3dcb87e.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-fb44a42088c9369a.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-61a7402e61a5b0e0.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-63f8356c87a0d0e8.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-a506e577d917828c.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-fc1fb63210fdafad.rlib" "-Wl,--end-group" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-c21be34a5cae8449.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/tmp/pip-wheel-0xu3npvl/cryptography_8a6adfad0ef1405098dc7122c13fc74c/src/rust/target/release/build/parking_lot_core-506a93a9972231b0/build_script_build-506a93a9972231b0" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-nodefaultlibs"
        = note: /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)
                /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)
                /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)
                /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)
                /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)

I now presume that this second option is the right path forward but I would need is the ability to set "host" environment variables so it doesn't use the default set of *FLAGS

EDIT : This is where using a target CMAKE configuration file makes it trivial as it then only refers to it for target-side builds, leaving host-side tools being built properly. I wasn't able to achieve a similar behavior using cargo/rustc with config.toml as it didn't seemed to be used/found. As such I ended-up using environment variables instead.

My guess is that I would need to specify for the build at the very least HOST_LDFLAGS= for linker options for host, and also perhaps:

  • HOST_CFLAGS= for C options for host.
  • HOST_CPPFLAGS= for C preprocessor options for host.

th0ma7 avatar Sep 20 '22 11:09 th0ma7

RUSTFLAGS="--target=aarch64-unknown-linux-gnu

I think this is problematic, --target should not be specified in RUSTFLAGS since RUSTFLAGS will applies to both build scripts (when no --target argument is passed to cargo build I think, https://github.com/rust-lang/cargo/issues/4423) and target so you're forcing aarch64 for host build.

messense avatar Sep 22 '22 08:09 messense

I think this is problematic, --target should not be specified in RUSTFLAGS since RUSTFLAGS will applies to both build scripts (when no --target argument is passed to cargo build I think, rust-lang/cargo#4423) and target so you're forcing aarch64 for host build.

Indeed, I took good note of that. And thnx for the link, it refers to another set of variable that I might have overseen such as CARGO_TARGET_<arch-triplet>_RUSTFLAGS which could be promising.

This thread also looks promising https://github.com/rust-lang/rust/issues/94003

Will further investigate that.

th0ma7 avatar Sep 22 '22 20:09 th0ma7

@alex got a fix for this that you might like to know for other issues like this one, its a question of setting --env LIBRARY_PATH=when creating thecrossenvwhich "unsets" the value of the offendinfLIBRARY_PATH` when cross-compiling rust wheels.

For reference: https://github.com/PyO3/setuptools-rust/issues/294#issuecomment-1264339712

Closing as issue is with setuptools-rust with using a workaround. Thnx anyway for the help.

th0ma7 avatar Oct 02 '22 22:10 th0ma7