cross "no matching manifest" on cross v0.2.5

Checklist

[X] I've looked through the issues and pull requests for similar reports

Describe your issue

I've been cross-compiling https://github.com/automerge/automerge-go for a while. Today I upgraded cross, and it broke cross-compilation:

$ cargo init --bin hello
$ cd hello
$ cross run --target aarch64-unknown-linux-gnu
Unable to find image 'ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/aarch64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.

The error message is the same as the one I get when I run docker with no --platform argument, when I do not have the image cached locally (once the image has run once, I no longer need to pass --platform to docker to make it work).

In case it helps tracking this down, I'm using Docker version 20.10.23, build 7155243 (as provided by Docker Desktop 4.17.0 (99724).

$ docker run -it  ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 /bin/bash
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.

$ docker run -it --platform linux/amd64 ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 /bin/bash
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
58690f9b18fc: Pull complete
b51569e7c507: Pull complete
da8ef40b9eca: Pull complete
fb15d46c38dc: Pull complete
7cd0932c59ac: Pull complete
b8099f45b9a5: Pull complete
640227c3e7fb: Pull complete
feac1a957a2e: Pull complete
2bb7cc963f55: Pull complete
69f3cca19ad2: Pull complete
9eae8e6ab0e2: Pull complete
f6948013d489: Pull complete
e63c3aabe8fe: Pull complete
f3f8758fddaa: Pull complete
e8df8713a13e: Pull complete
840b2ff81099: Pull complete
e52d8b455ef1: Pull complete
Digest: sha256:9e5b39c09874bc1816c675ed11afca2c2ed6cee0c4ed2b3c1d5763c346c9ae3f
Status: Downloaded newer image for ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5
root@ad6f0bc127c6:/#

docker run -it  ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 /bin/bash
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
root@9e3a9c7477c6:/# exit

What target(s) are you cross-compiling for?

aarch64-unknown-linux-gnu, x86_64-unknown-linux-gnu

Which operating system is the host (e.g computer cross is on) running?

[X] macOS
[ ] Windows
[ ] Linux / BSD
[ ] other OS (specify in description)

What architecture is the host?

[ ] x86_64 / AMD64
[ ] arm32
[X] arm64 (including Mac M1)

What container engine is cross using?

[X] docker
[ ] podman
[ ] other container engine (specify in description)

cross version

0.2.5

Example

$ cargo init --bin hello
$ cd hello
$ cross run --target aarch64-unknown-linux-gnu
Unable to find image 'ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/aarch64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.

Additional information / notes

This is not a problem on the previous version with cargo install [email protected]. Everything works fine!

Mar 02 '23 05:03 ConradIrwin

I think this is a regression in docker, pretty sure it used to work

$ cross build --target x86_64-unknown-linux-gnu -v
+ cargo metadata --format-version 1 --filter-platform x86_64-unknown-linux-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/local/bin/docker
+ /usr/local/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=emil' --rm --user 501:20 -v /Users/emil/.xargo:/xargo:z -v /Users/emil/.cargo:/cargo:z -v /cargo/bin -v /Users/emil/workspace/cross:/project:z -v /Users/emil/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:z,ro -v /Users/emil/workspace/cross/target:/target:z -w /project -i -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo build --target x86_64-unknown-linux-gnu -v'
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu

the fix is as mentioned adding --platform, which was added in https://github.com/cross-rs/cross/pull/817

for now, if you must use 0.2.5, use CROSS_CONTAINER_OPTS="--platform linux/amd64"

$ CROSS_CONTAINER_OPTS="--platform linux/amd64" cross build --target x86_64-unknown-linux-gnu -v
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/local/bin/docker
+ /usr/local/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=emil' --platform linux/amd64 --rm --user 501:20 -v /Users/emil/.xargo:/xargo:z -v /Users/emil/.cargo:/cargo:z -v /cargo/bin -v /Users/emil/workspace/cross:/project:z -v /Users/emil/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:z,ro -v /Users/emil/workspace/cross/target:/target:z -w /project -i -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo build --target x86_64-unknown-linux-gnu -v'
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
58690f9b18fc: Pulling fs layer 
b51569e7c507: Pulling fs layer 
da8ef40b9eca: Pulling fs layer 
fb15d46c38dc: Waiting 
7cd0932c59ac: Waiting 
b8099f45b9a5: Waiting 
640227c3e7fb: Waiting 
...

Mar 02 '23 16:03 Emilgardis

Hmm, actually, is it a change in the oci images?

$ cross-v0.2.5 metadata --target x86_64-unknown-linux-gnu -v
+ cargo metadata --format-version 1 --filter-platform x86_64-unknown-linux-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/local/bin/docker
+ /usr/local/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=emil' --rm --user 501:20 -v /Users/emil/.xargo:/xargo:z -v /Users/emil/.cargo:/cargo:z -v /cargo/bin -v /Users/emil/workspace/cross:/project:z -v /Users/emil/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:z,ro -v /Users/emil/workspace/cross/target:/target:z -w /project -i -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo metadata -v'
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu

$ cross-v0.2.4 metadata --target x86_64-unknown-linux-gnu -v
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/local/bin/docker
+ /usr/local/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=emil' --rm --user 501:20 -v /Users/emil/.xargo:/xargo:Z -v /Users/emil/.cargo:/cargo:Z -v /cargo/bin -v /Users/emil/workspace/cross:/project:Z -v /Users/emil/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:Z,ro -v /Users/emil/workspace/cross/target:/target:Z -w /project -i -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.4 sh -c 'PATH=$PATH:/rust/bin cargo metadata -v'
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.4' locally
0.2.4: Pulling from cross-rs/x86_64-unknown-linux-gnu
58690f9b18fc: Pulling fs layer 
b51569e7c507: Pulling fs layer 
da8ef40b9eca: Pulling fs layer 
fb15d46c38dc: Waiting 
ba4cfa0d02e9: Waiting 
9b3d6abb553f: Waiting 
7e4f4e01d8cc: Waiting 
7457ea83291e: Waiting 
385151f4cf4a: Waiting 
dd3a4857dad0: Waiting 
f4c64b50a179: Waiting 
634c7871468d: Waiting 
94489cc40d0c: Waiting 
f47e7b8372aa: Waiting 
70626baa3f44: Waiting 
8c9e9eeed2e0: Waiting 
de83618d0833: Waiting 
^C

there's no changes in the invocation between 0.2.4 and 0.2.5 except we're using the tag 0.2.5 and using :z on mounts

very strange

Mar 02 '23 17:03 Emilgardis

ehm, what's this?

unknown/unknown

Mar 02 '23 18:03 Emilgardis

the sha for unknown/unknown matches

https://github.com/cross-rs/cross/actions/runs/4092128923/jobs/7057264550#step:16:129


#23 exporting to image
#23 exporting layers done
#23 exporting manifest sha256:a2d924b0399f17432c3c8d6af2136903c5105bf71c726044a988427f849a8509 done
#23 exporting config sha256:77a3259dcb3ad2fec032d8d51dea3ae852ea488ff33a0261a3dc3d3a06f6b7fb done
#23 exporting attestation manifest sha256:a62738a88fd7403bf1fd5beb14d391013206ec8e7d3a4ba6522cf61a411b20a0 done
#23 exporting manifest list sha256:9e5b39c09874bc1816c675ed11afca2c2ed6cee0c4ed2b3c1d5763c346c9ae3f done
#23 pushing layers
#23 ...

I've never seen this before, but apparently it's https://docs.docker.com/build/attestations/attestation-storage/

not sure if it's worth disabling, since I definitely feel like this is an upstream issue in that, imo, if the manifest only has one valid and supported platform, it should use that, if there's multiple supported platforms, then prompt the user to pick.

Mar 02 '23 18:03 Emilgardis

related: https://github.com/docker/build-push-action/issues/820

we don't use the build-push-action, but I suspect the issue it the same here as we basically do the same thing as it.

Mar 02 '23 18:03 Emilgardis

So to summarize, the problem is

cross <0.3.0 calls docker run --rm -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:$version, no --platform specified
the image for :0.2.5 was built with docker buildx build, on a linux/amd64 system where provenence/attestation is supported by buildx/buildkit (moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.11.2 944939944ca4cc58a11ace4af714083cfcd9a3c7). No --platform specified
if provenence is used the manifest becomes multi-arch, always including unknown/unknown for the build attestations and then other platforms that was included in the build. 0.2.5 manifest is

$ docker buildx imagetools inspect ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5
Name:      ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:9e5b39c09874bc1816c675ed11afca2c2ed6cee0c4ed2b3c1d5763c346c9ae3f
           
Manifests: 
  Name:        ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5@sha256:a2d924b0399f17432c3c8d6af2136903c5105bf71c726044a988427f849a8509
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64
               
  Name:        ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5@sha256:a62738a88fd7403bf1fd5beb14d391013206ec8e7d3a4ba6522cf61a411b20a0
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.type:   attestation-manifest
    vnd.docker.reference.digest: sha256:a2d924b0399f17432c3c8d6af2136903c5105bf71c726044a988427f849a8509

docker run and docker pull, when a manifest is a single platform, will try to use that platform, if it's multi-arch, it'll use the same platform as the default, and if it doesn't exist panic.

now, there's one issue, we never enabled provenence. https://github.com/docker/build-push-action/issues/771 suggests that setting --provence=false would fix this, but the docs for buildx 0.11 suggest that that is the default. I'm very confused...

edit: ah, https://docs.docker.com/engine/reference/commandline/buildx_build/#provenance

By default, a minimal provenance attestation will be created for the build result, which will only be attached for images pushed to registries.

as can be seen here https://github.com/cross-rs/cross/actions/runs/4092128923/jobs/7057264550#step:10:5997

#21 exporting to docker image format
#21 exporting layers
#21 exporting layers 31.6s done
#21 exporting manifest sha256:3fa66b56f1735512fe1ce052aa69e94b019c6beb32e06114695ae48f266ee12b done
#21 exporting config sha256:061c8657a90b4c9588ac4159f6b35526c705e8d5e8d1dcded9e5bb35c2effc6d done
#21 sending tarball
#21 ...

Mar 02 '23 19:03 Emilgardis

so there's three ways to solve this

specify --provenence=false, rebuild the images and publish
specify buildkit version = 0.10, rebuild images and publish
publish cross 0.2.6 with a fix including --platform=linux/amd64, this should probably also include https://github.com/cross-rs/cross/pull/1142

Mar 02 '23 20:03 Emilgardis

Thanks for figuring this out! There's no rush on a fix from my point of view (I can use 0.2.4 successfully).

Mar 02 '23 23:03 ConradIrwin

If you missed it, you can also use CROSS_CONTAINER_OPTS="--platform linux/amd64" to tell docker that you actually want to use the amd64 image.

use CROSS_BUILD_OPTS="--platform linux/amd64" for pre-build

cross on main already passes --platform

Mar 02 '23 23:03 Emilgardis

For anybody else, to save you spending days looking for a solution here is how I solved it: cargo install cross --git https://github.com/cross-rs/cross Yep, update and it worked fine for me.

Apr 19 '23 10:04 ActuallyHappening

Hey, sorry for my lack of knowledge, but what's exactly the problem that blocks cross of having arm native docker images?

Dec 29 '23 17:12 CosminPerRam

Hey, sorry for my lack of knowledge, but what's exactly the problem that blocks cross of having arm native docker images?

The blocker is implementation in https://github.com/cross-rs/cross/issues/975 and particularly access to GHA runners on aarch64 architecture to not have to emulate the containers in CI

Dec 29 '23 21:12 Emilgardis