buildkit Strange $PATH searching with qemu

I use buildkit and contrainerd to build the image. The host is aarch64 (raspberry pi 4B 8G).

I build an image for amd64 with:

buildctl build --frontend dockerfile.v0 --local dockerfile=. --local context=. --output type=image,name=xxx,push=true --secret id=docker,src=xxx.json --opt platform=linux/amd64 --no-cache --progress plain

But get the output:

If I add /aa/bb/ to the beginning of $PATH,

If I build for arm64 with:

buildctl build --frontend dockerfile.v0 --local dockerfile=. --local context=. --output type=image,name=xxx,push=true --secret id=docker,src=xxx.json --opt platform=linux/arm64 --no-cache --progress plain

Other infos:

$ uname -a
Linux fulgens 5.3.18-150300.59.68-default #1 SMP Wed May 4 11:29:09 UTC 2022 (ea30951) aarch64 aarch64 aarch64 GNU/Linux

$ binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/riscv64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/mips64le",
    "linux/mips64",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-i386",
    "qemu-mips64",
    "qemu-mips64el",
    "qemu-ppc64le",
    "qemu-riscv64",
    "qemu-s390x",
    "qemu-x86_64"
  ]
}

$ buildctl --version
buildctl github.com/moby/buildkit v0.10.4 a2ba6869363812a210fcc3ded6926757ab780b5f

$ binfmt --version
binfmt/1aa2eba qemu/v6.2.0 go/1.17.8

Aug 24 '22 09:08 Qinka

There is another example, build for the platform linux/amd64 on linux/arm64 host (failed). It is success on linux/amd64 host to build the platform linux/amd64.

The output of failed one:

#14 266.9   |
#14 266.9   = note: "cc" "-m64" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/rcrt1.o" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crti.o" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtbeginS.o" "/tmp/rustcOrRiGa/symbols.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.0.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.1.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.10.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.11.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.12.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.13.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.14.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.15.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.2.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.3.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.4.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.5.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.6.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.7.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.8.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.build_script_build.520a45f9-cgu.9.rcgu.o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d.4gt7y29bl88tb0v8.rcgu.o" "-Wl,--as-needed" "-L" "/src/target/release/deps" "-L" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-ae5a1f9055bd29a2.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-e6fb41ea71380a2f.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libobject-b2c406136087fc2d.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libmemchr-fa317be2c1b9799a.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libaddr2line-3199c88d989d542c.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libgimli-7c36f66a661248f4.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-c26ec917f656f77c.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_detect-b128130c787a7cc5.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-59681a8bfbafa934.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libminiz_oxide-d445f5bc429e1254.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libadler-d5b729dab5ac2d91.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-4c631df2c61c7e45.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-a25e052eb4c048f3.rlib" "-lunwind" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-c811ebcf2c7483ce.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-4437308e4067e2d6.rlib" "-lc" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-7006e8f8af159628.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-1de74813601370a9.rlib" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-02064f71b04e39be.rlib" "-Wl,--end-group" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-025a46b1683d0984.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-nostartfiles" "-L" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-L" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained" "-o" "/src/target/release/build/serde_derive-27f000cfccc0fd0d/build_script_build-27f000cfccc0fd0d" "-Wl,--gc-sections" "-static-pie" "-Wl,-zrelro,-znow" "-nodefaultlibs" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtendS.o" "/usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtn.o"
#14 266.9   = note: Error while loading /usr/local/rustup/toolchains/1.63.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin/cc: No such file or directory

PATH and cc:

#6 [builder 2/8] RUN echo /usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#6 0.331 /usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#6 DONE 0.4s

#9 [builder 3/8] RUN which -a cc
#9 0.389 /usr/bin/cc
#9 DONE 0.5s

Sep 05 '22 16:09 Qinka

I'm running into this same issue when trying to use pyenv in my build. It works fine on the M1 Mac running arm64, but fails when the build is being emulated. Lots of weird path searching. As @Qinka described, it seems to check the first element in PATH, and doesn't check the remaining elements.

Nov 29 '22 16:11 mattvaughan

It looks like there may be some solutions in this thread docker/buildx#495

Nov 29 '22 17:11 mattvaughan

Should be fixed with https://github.com/tonistiigi/binfmt/pull/114

I pushed the binfmt images:

crazymax/binfmt:fix-execvp-path
crazymax/binfmt:fix-execvp-path-buildkit

And also a BuildKit image using crazymax/binfmt:fix-execvp-path-buildkit, if you want to check if everything looks good on your side:

crazymax/buildkit:fix-execvp-path (based on BuildKit v0.10.6)

To create a container builder with Buildx using this image:

$ docker buildx create --name mybuilder --driver-opt image=crazymax/buildkit:fix-execvp-path --use

Nov 29 '22 17:11 crazy-max