Cross CPU architecture compilation

[qdm12]

• [ ] Dockerfile changed to be buildable on all platforms supported by Docker
  • [x] amd64, 386, arm64, armv7, armv6
  • [ ] Wait for the Rust standard library to support s390x, ppc64le and riscv64
  • [ ] Wait for Alpine to support riscv64
• [x] CI changed to build for amd64, 386, arm64, armv6 and armv7
• [x] Build static binaries by bunding musl (+3MB)
• [x] Try dynamically building for musl to have a smaller Alpine final Docker image instead of building static binaries:
  • Only saves 500KB on a 6MB binary, so not worth the trouble
  • Can be set by setting --build-arg STATIC=no when building the image
• [x] ~Compilation might need to be changed depending on https://github.com/rust-lang/compiler-team/issues/422~ No, see above. In the worst, case it will still pass as the final image is Alpine which has musl. Maybe the static check will fail to pass, but that's an easy fix.

[logopk]

Good morning,

I'm having a hard time getting this to compile for armv7 on a raspberrypi 3b+. Even though I'm using docker build it's just simply not configured to use armv7 options. As my wireguard is running on that pi, I have no other option...

I would really appreciate to get some hints to configure rust in the docker build for armv7. ( I am an absolute rust newb)

Currently it fails in ring

   Compiling ring v0.16.20
error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/usr/src/prometheus_wireguard_exporter/target/release/build/ring-8f657e0472a3585e/build-script-build` (exit code: 101)
  --- stdout
  OPT_LEVEL = Some("3")
  TARGET = Some("armv7-unknown-linux-musleabihf")
  HOST = Some("armv7-unknown-linux-gnueabihf")
  CC_armv7-unknown-linux-musleabihf = None
  CC_armv7_unknown_linux_musleabihf = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_armv7-unknown-linux-musleabihf = None
  CFLAGS_armv7_unknown_linux_musleabihf = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = None

  --- stderr
  running "arm-linux-musleabihf-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-march=armv7-a" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/usr/src/prometheus_wireguard_exporter/target/armv7-unknown-linux-musleabihf/release/build/ring-8914cac724f6beee/out/aesv8-armx-linux32.o" "/usr/local/cargo/registry/src/github.com-1285ae84e5963aae/ring-0.16.20/pregenerated/aesv8-armx-linux32.S"
  thread 'main' panicked at 'failed to execute ["arm-linux-musleabihf-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-march=armv7-a" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/usr/src/prometheus_wireguard_exporter/target/armv7-unknown-linux-musleabihf/release/build/ring-8914cac724f6beee/out/aesv8-armx-linux32.o" "/usr/local/cargo/registry/src/github.com-1285ae84e5963aae/ring-0.16.20/pregenerated/aesv8-armx-linux32.S"]: No such file or directory (os error 2)', /usr/local/cargo/registry/src/github.com-1285ae84e5963aae/ring-0.16.20/build.rs:653:9

Apparently the CC-vars are not populated and exporting then (docker ENV) does not fill them. this is discussed on https://github.com/briansmith/ring

Besides this the 1-alpine3.12 image does not exist for armv7. So I had to move to buster.

+ARG ARCH=arm
+RUN apt update && apt install -y  musl-dev
+RUN rustup target add armv7-unknown-linux-musleabihf
...
+RUN cargo install --target armv7-unknown-linux-musleabihf --path .

Thank you!

[qdm12]

Quite very late for a reply, but I made #63 to cross build Docker images for all ARM flavors, which should work. You should be able to pull the image for your architecture once it's merged in the master branch.

I have updated the checkboxes above on things to do, let's not close the issue yet.

[qdm12]

Ooof that took me a solid day of work, god rust isn't that friendly with cross compilation! 😢 But here we are, you can try the image mindflavor/prometheus-wireguard-exporter:multi-arch-dockerfile which is now built for amd64, 386, arm64, armv6 and armv7. There are still some other niche platform missing but this is due to Rust's standard library support lacking.

[logopk]

Thanks @qdm12.

Unfortunately the application does not start. It exits without output.

Tried also with docker run --rm mindflavor/prometheus-wireguard-exporter:multi-arch-dockerfile

Anything I can do to debug on armv7 (pi 3b+)?

[qdm12]

Ooops my bad, my dependency caching mechanism was instead building an empty program main() {} 😄

It's now fixed, although it's still building here.

Actually, if you don't mind, can you try running (you need git installed):

docker build -t mindflavor/prometheus_wireguard_exporter https://github.com/MindFlavor/prometheus_wireguard_exporter.git#multi-arch-dockerfile

On your ARM device see if it works? It might take a good 10 minutes I think to build though.

If it doesn't work try preceeding the command with DOCKER_BUILDKIT=1 ? Thanks!

[logopk]

It works - I did wait for your build though, as my pi was overwhelmed by the build....

Thanks a lot for this!

[qdm12]

Cool 👍 Actually just as a note, you cannot build the image if your host is not amd64 or 386 so it won't work on ARM devices anyway 😢