Cannot cross compile (host: macos, target: x86_64-unknown-linux-musl or x86_64-unknown-linux-gcc)

[hgrecco]

When trying to cross compile ring, I get the following:

host: macos target: x86_64-unknown-linux-musl

error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `[...]/target/release/build/ring-4c4abdebf03c0195/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("3")
  TARGET = Some("x86_64-unknown-linux-musl")
  HOST = Some("x86_64-apple-darwin")
  CC_x86_64-unknown-linux-musl = None
  CC_x86_64_unknown_linux_musl = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_x86_64-unknown-linux-musl = None
  CFLAGS_x86_64_unknown_linux_musl = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")

  --- stderr
  running "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-std=c1x" "-Wbad-function-cast" "-Wnested-externs" "-Wstrict-prototypes" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/Users/username/Documents/code/anotrs/target/x86_64-unknown-linux-musl/release/build/ring-6bdb0d88143b94e6/out/aes_nohw.o" "crypto/fipsmodule/aes/aes_nohw.c"
  thread 'main' panicked at 'failed to execute ["musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-std=c1x" "-Wbad-function-cast" "-Wnested-externs" "-Wstrict-prototypes" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/Users/username/Documents/code/anotrs/target/x86_64-unknown-linux-musl/release/build/ring-6bdb0d88143b94e6/out/aes_nohw.o" "crypto/fipsmodule/aes/aes_nohw.c"]: No such file or directory (os error 2)', /Users/username/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:653:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

and

host: macos target: x86_64-unknown-linux-gcc

error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/Users/username/Documents/code/anotrs/target/release/build/ring-4c4abdebf03c0195/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("3")
  TARGET = Some("x86_64-unknown-linux-gnu")
  HOST = Some("x86_64-apple-darwin")
  CC_x86_64-unknown-linux-gnu = None
  CC_x86_64_unknown_linux_gnu = None
  TARGET_CC = Some("clang")
  CFLAGS_x86_64-unknown-linux-gnu = None
  CFLAGS_x86_64_unknown_linux_gnu = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")

  --- stderr
  running "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=x86_64-unknown-linux-gnu" "-I" "include" "-Wall" "-Wextra" "-std=c1x" "-Wbad-function-cast" "-Wnested-externs" "-Wstrict-prototypes" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-DNDEBUG" "-c" "-o/Users/username/Documents/code/anotrs/target/x86_64-unknown-linux-gnu/release/build/ring-37bfac9d69f0e178/out/aes_nohw.o" "crypto/fipsmodule/aes/aes_nohw.c"
  In file included from crypto/fipsmodule/aes/aes_nohw.c:17:
  In file included from crypto/fipsmodule/aes/../../internal.h:114:
  include/GFp/check.h:27:11: fatal error: 'assert.h' file not found
  # include <assert.h>
            ^~~~~~~~~~
  1 error generated.
  thread 'main' panicked at 'execution failed', /Users/username/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:656:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I am using a this as a workaround https://hub.docker.com/r/ekidd/rust-musl-builder/ but it seems to be much slower. Any idea how to fix the above mentioned errors?

[briansmith]

I have never tried to cross compile from macOS to another OS. However, I suggest you look at how we compile to -musl targets in CI using clang, and then try to use clang (probably clang downloaded from LLVM, not Apple clang) to try the cross-compilation.

[taind772]

Edit: Pardon me, i forgot to install musl package I bumped into the same problem in Manjaro(linux-5.10.60 - clang 12.0.1) too

error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `[...]/target/release/build/ring-e3ac515dcadd1e70/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("3")
  TARGET = Some("x86_64-unknown-linux-musl")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_x86_64-unknown-linux-musl = None
  CC_x86_64_unknown_linux_musl = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_x86_64-unknown-linux-musl = None
  CFLAGS_x86_64_unknown_linux_musl = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")

  --- stderr
  running "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/[...]/target/x86_64-unknown-linux-musl/release/build/ring-d2b907aa8e6446d7/out/aesni-x86_64-elf.o" "/home/ndt/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S"
  thread 'main' panicked at 'failed to execute ["musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o[...]/target/x86_64-unknown-linux-musl/release/build/ring-d2b907aa8e6446d7/out/aesni-x86_64-elf.o" "/home/ndt/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S"]: No such file or directory (os error 2)', /home/ndt/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:653:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[loafofpiecrust]

I'm having similar issues cross-compiling from darwin to x86_64-unknown-linux-gnu, and this is blocking me from building my project on one of my dev machines. My compilation uses clang during the ring build and I get different errors:

error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/private/tmp/nix-build-rust-workspace-deps-unknown.drv-1/dummy-src/target/release/build/ring-2aa05c1c87c692bc/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("3")
  TARGET = Some("x86_64-unknown-linux-gnu")
  HOST = Some("x86_64-apple-darwin")
  CC_x86_64-unknown-linux-gnu = None
  CC_x86_64_unknown_linux_gnu = None
  TARGET_CC = None
  CC = Some("clang")
  CFLAGS_x86_64-unknown-linux-gnu = None
  CFLAGS_x86_64_unknown_linux_gnu = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")

  --- stderr
  running "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=x86_64-unknown-linux-gnu" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align"
  /nix/store/spiwww56pgyq770mznavcykphs2nynkf-crates-io/ring-0.16.20/pregenerated/aesni-x86_64-elf.S:13:1: error: unknown directive
  .hidden GFp_ia32cap_P
  ^
  /nix/store/spiwww56pgyq770mznavcykphs2nynkf-crates-io/ring-0.16.20/pregenerated/aesni-x86_64-elf.S:15:1: error: unknown directive
  .hidden GFp_aes_hw_encrypt
  ^

Any ideas?

[loafofpiecrust]

Looks like if I set TARGET_CC=x86_64-unknown-linux-gnu-gcc then I get a different set of errors because ring is compiling specifically for MacOS, rather than my target platform! 'Framework' seems like a MacOS-only dependency.

86_64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-iframework'

[CyanBlob]

I've got the same problem building for any of:

• arm-unknown-linux-gnueabihf
• arm-unknown-linux-musleabi
• armv7-unknown-linux-gnueabihf
• armv7-unknown-linux-musleabihf

Error:

error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/Users/andrew/Development/duolingo-word-of-the-day/target/debug/build/ring-6af67c4412ae572f/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("0")
  TARGET = Some("armv7-unknown-linux-musleabihf")
  HOST = Some("aarch64-apple-darwin")
  CC_armv7-unknown-linux-musleabihf = None
  CC_armv7_unknown_linux_musleabihf = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_armv7-unknown-linux-musleabihf = None
  CFLAGS_armv7_unknown_linux_musleabihf = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = None

  --- stderr
  running "arm-linux-musleabihf-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-march=armv7-a" "-mfpu=vfpv3-d16" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/Users/andrew/Development/duolingo-word-of-the-day/target/armv7-unknown-linux-musleabihf/debug/build/ring-516f151c2cbc25e3/out/aesv8-armx-linux32.o" "/Users/andrew/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesv8-armx-linux32.S"
  thread 'main' panicked at 'failed to execute ["arm-linux-musleabihf-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-march=armv7-a" "-mfpu=vfpv3-d16" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/Users/andrew/Development/duolingo-word-of-the-day/target/armv7-unknown-linux-musleabihf/debug/build/ring-516f151c2cbc25e3/out/aesv8-armx-linux32.o" "/Users/andrew/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesv8-armx-linux32.S"]: No such file or directory (os error 2)', /Users/andrew/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:653:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[CyanBlob]

I can work around this via Docker like:

docker pull messense/rust-musl-cross:armv7-musleabihf
alias rust-musl-builder='docker run --rm -it -v "$(pwd)":/home/rust/src messense/rust-musl-cross:armv7-musleabihf'
rust-musl-builder cargo build --release

Just in case anybody else gets stuck! I couldn't get the Docker container hgrecco recommended working for other reasons, but this works well enough. For the first gen Pi Zero and possibly the standard Pi 1, remove the "v7" from the image tag in both places it occurs

[briansmith]

• arm-unknown-linux-gnueabihf
• arm-unknown-linux-musleabi
• armv7-unknown-linux-gnueabihf
• armv7-unknown-linux-musleabihf

The ring CI/CD does build for some of those platforms, and shows how to install the tools to do so, and how to invoke the build with the right environment variables. The only catch is that ring's CI/CD is doing that from a Linux host, but the same principles apply.

When using clang on macOS, make sure you're using Clang downloaded from LLVM and not the Apple Clang.

Also make sure you set TARGET_CC and use --target on the cargo command line. I believe you have to set TARGET_AR as well. Again, this is all documented (in executable code) in /.github/ci.yml and mk/*.sh.

I intend to better document the cross-compilation of ring on a macOS host, and automate away some of the tedium. It would be useful for somebody who already has it working to explain how, instead of me having to figure it out. I do build stuff on macOS, but never cross-compiling to Linux.

[CyanBlob]

@briansmith Thanks for the tip! I got it building.

For those that are curious, here's what I did:

Install Clang from homebrew (this is the LLVM Clang, not Apple Clang)

brew install llvm

Set up environment:

export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"
export TARGET_CC=$(which clang)

Install linker

brew install arm-linux-gnueabihf-binutils These tools are for the Raspberry Pi Zero; make sure to use the correct target for your hardware

Add Rust toolchain

rustup target add arm-unknown-linux-musleabihf

Set linker in .cargo/config

[target.arm-unknown-linux-musleabihf]
linker = "arm-linux-gnueabihf-ld"

These settings are for the Raspberry Pi Zero; make sure to use the correct target for your hardware

Build

cargo build --target=arm-unknown-linux-musleabihf

If all goes well, the binary should be placed in target/arm-unknown-linux-musleabihf/debug

I think this should be more or less comprehensive. If anybody tries and can't get it to work feel free to ask me!

[ghost]

@CyanBlob I've tried your steps, trying to cross-compile for x86_64-unknown-linux-gnu

setting

export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"
export TARGET_CC=$(which clang)

after brew install llvm does the trick with the build, but if fails later down the pipeline

error: could not compile `test` due to previous error;
ld: unknown option: --as-needed

I'm guessing it's because I haven't changed the linker as you, but I can't find anything for x86_64-unknown-linux-gnu on homebrew. I've tried with x86_64-elf-binutils without success:

error: ...
  = note: x86_64-elf-ld: cannot find -lgcc_s: No such file or directory
          x86_64-elf-ld: cannot find -lutil: No such file or directory
          x86_64-elf-ld: cannot find -lrt: No such file or directory
          x86_64-elf-ld: cannot find -lpthread: No such file or directory
          x86_64-elf-ld: cannot find -lm: No such file or directory
          x86_64-elf-ld: cannot find -ldl: No such file or directory
          x86_64-elf-ld: cannot find -lc: No such file or directory

Any idea on how I may proceed here?

[briansmith]

After I address #1473, it should be pretty easy to extend that work so that we don't need a C toolchain for cross-compiling to Linux.

[valarauca]

I also get this error while compiling 0.16.20 from AMD64-GNU-Linux for AMD64-MUSL-Linux.

Setting TARGET_CC does not change the error received.