bytecodealliance
Unable to link a binary with wasmtime and musl
Test Case
I'm sorry I cannot find an isolated example for you to test with a small dependency tree, but in our repo, when we updated wasmtime from 21.0.1 to 22.0.0 we suddenly started having issues linking the binary when compiling against musl target.
Here's the branch:
https://github.com/grafbase/grafbase/pull/1840
And the error we see when linking:
= note: /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(e_pow.o): warning: relocation against `_dl_x86_cpu_features' in read-only section `.text'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_ceil.o): in function `__ceil_ifunc':
(.text+0x6): undefined reference to `_dl_x86_cpu_features'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_trunc.o): in function `__trunc_ifunc':
(.text+0x6): undefined reference to `_dl_x86_cpu_features'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_fma.o): in function `__fma_ifunc':
(.text+0xafd): undefined reference to `_dl_x86_cpu_features'
/usr/bin/ld: (.text+0xb06): undefined reference to `_dl_x86_cpu_features'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(w_fmod.o): in function `__fmod':
(.text+0x41): undefined reference to `errno'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(w_pow.o): in function `__pow':
(.text+0x75): undefined reference to `errno'
/usr/bin/ld: (.text+0xab): undefined reference to `errno'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_truncf.o): in function `__truncf_ifunc':
(.text+0x6): undefined reference to `_dl_x86_cpu_features'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(math_err.o): in function `with_errno':
(.text+0x3): undefined reference to `errno'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(math_err.o): in function `with_errno.constprop.0':
(.text+0x13): undefined reference to `errno'
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(e_pow.o): in function `__ieee754_pow_ifunc':
(.text+0x746): undefined reference to `_dl_x86_cpu_features'
/usr/bin/ld: (.text+0x74f): undefined reference to `_dl_x86_cpu_features'
/usr/bin/ld: (.text+0x75f): undefined reference to `_dl_x86_cpu_features'
/usr/bin/ld: read-only segment has dynamic relocations
collect2: error: ld returned 1 exit status
We can currently work with the previous version, but this issue prevents us from updating to the latest runtime.
Steps to Reproduce
- fetch the branch linked in the previous chapter
- Make sure to have musl target installed from rustup
- Install musl-gcc if needed
cargo build -p grafbase-gateway --release --target x86_64-unknown-linux-musl
Expected Results
The project compiles correctly into a working binary.
Actual Results
Linking fails as explained in the first chapter.
Versions and Environment
Wasmtime version or commit: 22.0.0
Operating system: NixOS unstable and Debian Bookworm
Architecture: x86_64-unknown-linux-musl
Extra Info
This is most probably some issue with wasmtime and some other crate together. I could not reduce a nice and clean repro as a new project. Been trying to get that the whole afternoon... But something definitely broke in the latest update of wasmtime.
For one reason or another the linker is using glibc's version of libm rather than musl's version of libm. I'm not sure why that happens,. In any case you shouldn't need musl installed on the host system, nor use musl-gcc. Rustc already ships with a copy of musl's libc.a which includes libm.
This might be it. I installed a clean debian container to test this out and some dependency didn't build due to missing musl-gcc... I will get back to this when I'm on my computer again.
So, this is what I found out:
- wasmtime depends on ittapi, which depends on ittapi-sys
- ittapi-sys depends on cc crate, and fails to compile on a system without musl-gcc
- if installing musl-tools, the crate compiles but with version 22 of wasmtime the linking stage fails to the error described in the issue
I wonder how common these kind of issues are with musl? We do have several sys dependencies which contain C code and which compile just fine against musl. But why did the wasmtime 22 update break the linking?
I did a git-bisect today trying to find which commit broke the build and I think I found it:
https://github.com/bytecodealliance/wasmtime/commit/cc93e8f7f0476c89e1a2ca3a27518e0a7c4177aa#diff-0edccdae9479a6e6d11da47ad26d419b88045e14c5a6086ce7cdd8e236132ed4R38-R41
Ping @alexcrichton
Removing these lines from build.rs fixes the issue. But this probably breaks something else...
I agree with @bjorn3 that this error is probably a misconfigured linker error rather than anything else. Are you configuring the linker as used by Cargo for the musl target? For example CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER=musl-gcc. Usage of musl-gcc is auto-configured by the cc crate but the Rust compiler itself does not leverage the same decision making process so the Rust compiler uses cc by default which doesn't work with musl objects since then it crosses the wires by accident.
Hey @alexcrichton, is there some documentation how exactly the static musl compilation of a Rust project with C dependencies should look like? In general everybody seems to say you just add a target with rustup and add the target to the cargo cli params. This has worked in the past eight years I've been working with Rust quite well. Now, when changing the CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER to musl-gcc, immediately when getting a crate depending on libc we get:
error: linking with `cc` failed: exit status: 1
|
= note: LC_ALL="C" PATH="/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/pimeys/code/grafbase/grafbase/cli/.cargo/bin:/home/pimeys/code/grafbase/grafbase/node_modules/.bin:/nix/store/628y2lfy59bz200rcpbrh9sc2p9v83dm-cargo-insta-1.33.0/bin:/nix/store/vcpa0lb46lyk6afvaj1bqk6z3sayz4ak-cargo-nextest-0.9.72/bin:/nix/store/pxnxx19lr32mqgxkpsz4h7qixzyd84q8-cargo-component-0.13.2/bin:/nix/store/9dna4y1s34nv8glc6d7pkp3a0a9v6ijr-hey-0.1.4/bin:/nix/store/z8lzlvwbygb865qx25wrvxpfa9x8k81i-cargo-bloat-0.12.1/bin:/nix/store/fdg2c6w444j2718mi6q88n5g4lap51pa-cargo-make-0.37.12/bin:/nix/store/q0v146ks60hyw0lc9z91gg7b991rsvys-cargo-release-0.25.0/bin:/nix/store/gfsnb85cmpavq0qkxl0k62jrxmvrriz8-npm-10.8.1/bin:/nix/store/kkkk3zjfa8nm6ghdd1rpdbavzvlyixcy-semver-7.6.2/bin:/nix/store/gcsghgcyax8ri9z9hif5cgqzqvq2dib0-sd-1.0.0/bin:/nix/store/6g9n96qf1yx139xklnmy3v4xhjvjgsji-nodejs-20.12.2/bin:/nix/store/v3ws3vn47r0lm16rjr6yriq6ywv4xvwv-prettier-3.3.0/bin:/nix/store/3ws1qnjgq1rbxs96q67php6i8aygcxgg-bun-1.1.10/bin:/nix/store/qannz09m66qpcy3ny1f4nkl4ql0g71ks-openssl-3.0.13-bin/bin:/nix/store/zdvrzlvzbn9ymb0z8na50w995j8np16z-pkg-config-wrapper-0.29.2/bin:/nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin:/nix/store/y74bzy6hjnc2n5d3hgq309imwfa0fpw4-rustup-1.26.0/bin:/nix/store/yrmlzr241pa9irf54fslilijnwv664c6-musl-1.2.3-dev/bin:/nix/store/bqjb40djywqk5n4dfzs5babpz7mycz04-musl-1.2.3-bin/bin:/nix/store/s25mgibyiwc2ahwpz2pbvdpj4vvngsva-pnpm-9.1.1/bin:/nix/store/k0d34jsjxzr7mg1c02vvbr616rb1m8nq-yarn-1.22.22/bin:/nix/store/hv6lq8yxrk5cb24147qaakp8isqq4pd9-cargo-about-0.6.2/bin:/nix/store/nbad47q0m0m9c5xid7zh05hiknwircbp-patchelf-0.15.0/bin:/nix/store/9bv7dcvmfcjnmg5mnqwqlq2wxfn8d7yi-gcc-wrapper-13.2.0/bin:/nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/bin:/nix/store/c2i631h8i5vcs1sqifwxfsazhwrg6wr5-glibc-2.39-52-bin/bin:/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/kln7kinji3b7sz8r50h4gn9yy6k1js9a-binutils-wrapper-2.41/bin:/nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin:/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/jjcsr5gs4qanf7ln5c6wgcq4sn75a978-findutils-4.9.0/bin:/nix/store/i34mknsjgrfyy71k2h79gda0bvagzc2j-diffutils-3.10/bin:/nix/store/5zjms21vpxlkbc0qyl5pmj2sidfmzmd7-gnused-4.9/bin:/nix/store/28gpmx3z6ss3znd7fhmrzmvk3x5lnfbk-gnugrep-3.11/bin:/nix/store/8vvkbgmnin1x2jkp7wcb2zg1p0vc4ks9-gawk-5.2.2/bin:/nix/store/rik7p68cq7yzlj5pmfpf4yv6jnrpvlgf-gnutar-1.35/bin:/nix/store/j5chw7v1x3vlmf3wmdpdb5gwh9hl0b80-gzip-1.13/bin:/nix/store/mxcq77rlan82dzpv3cgj0fh6qvv8ncil-bzip2-1.0.8-bin/bin:/nix/store/cdzpn0rdq810aknww3w9fy3wmw9ixr66-gnumake-4.4.1/bin:/nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin:/nix/store/0lfxbmchigx9vs9qmrlbahcy6nxwfnj1-patch-2.7.6/bin:/nix/store/6i4xxaa812vsbli9jkq4mksdddrk27lw-xz-5.4.6-bin/bin:/nix/store/xx7x1dwybpssfhq8yikvzz38bh3yrq97-file-5.45/bin:/home/pimeys/code/grafbase/grafbase/.direnv/bin:/home/pimeys/.cargo/bin:/home/pimeys/.local/bin:/home/pimeys/.atuin/bin:/run/wrappers/bin:/home/pimeys/.nix-profile/bin:/nix/profile/bin:/home/pimeys/.local/state/nix/profile/bin:/etc/profiles/per-user/pimeys/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcmngWD1/symbols.o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a.build_script_build.6d0fdb83d6651bae-cgu.0.rcgu.o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a.441cvjjahr7ngc3d.rcgu.o" "-Wl,--as-needed" "-L" "/home/pimeys/code/grafbase/grafbase/target/release/deps" "-L" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d5189b81a4fa4d36.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-6a4779412a873200.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-c88c426dd6780435.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-e74540b31113a555.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-8751b61bd13c15cf.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-a8b99dba9f449259.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-f0068d76172a0372.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-c16dfaf47799564e.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-bfa26dd63e299db5.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-75a2330a693e738f.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-76779dce1f7ab63d.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-01c5cc588623cb35.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-07bb9745ec737292.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-6eacdcc91004cefb.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-ae2488b58226c836.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-52acaddcaaba04c6.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-326b78eac9ecd050.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-307ebf19f0f13d30.rlib" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-d9076ee5964191bf.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/pimeys/code/grafbase/grafbase/target/release/build/serde-7224d35ce8b6f84a/build_script_build-7224d35ce8b6f84a" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
= note: /nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: /home/pimeys/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d5189b81a4fa4d36.rlib(std-d5189b81a4fa4d36.std.3c8ba8ebcf555201-cgu.0.rcgu.o): undefined reference to symbol 'gnu_get_libc_version@@GLIBC_2.2.5'
/nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libc.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
I'm all in on changing our build setup (probably even writing a blog post about that), but first I'd really like to know what is the recommended setup with musl? What we see with wasmtime is the added build param to include libm really started causing trouble. Without having that include, the compilation works just fine.
Unfortunately the linker configuration didn't seem to apply, in the error message you can see it's still executing cc. The solution is to get that plumbed correctly so rustc invokes musl-gcc as a linker instead of cc.
What we see with wasmtime is the added build param to include libm really started causing trouble
Cross compliation is not always easy or obvious. If your setup was executing cc before then it was incorrect. If it happened to work without -lm then that's basically just by luck. Cross compilation requires understanding error messages that are totally unrelated to the actual problem at hand unfortunately, such as this one.
Ok, fair. I think the plan is to just utilize Alpine and not bother with cross compilation. Did a full build using docker for the project and it spat out a fully functional binary without any issues.