Publish multi-platform images

[reuben]

trafficstars

It'd be great if the project could publish multi-platform images so that one can do multi-arch Docker builds using buildx and e.g. docker buildx build --platforms linux/amd64,linux/arm64.

See some relevant docs:

• https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/
• https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/

Right now, because the images for different architectures have different names, one either has to create a manifest by hand, or use separate downstream Dockerfiles for each arch. I've tried out with my own manifest and it works well for multi-platform builds:

$ docker pull quay.io/pypa/manylinux_2_24_aarch64:2022-03-31-361e6b6
$ docker pull quay.io/pypa/manylinux_2_24_x86_64:2022-03-31-361e6b6
$ docker tag quay.io/pypa/manylinux_2_24_x86_64:2022-03-31-361e6b6 ghcr.io/reuben/manylinux_2_24_x86_64:2022-03-31-361e6b6
$ docker tag quay.io/pypa/manylinux_2_24_aarch64:2022-03-31-361e6b6 ghcr.io/reuben/manylinux_2_24_aarch64:2022-03-31-361e6b6
$ docker login ghcr.io
$ docker push ghcr.io/reuben/manylinux_2_24_x86_64:2022-03-31-361e6b6
$ docker push ghcr.io/reuben/manylinux_2_24_aarch64:2022-03-31-361e6b6
$ docker manifest create ghcr.io/reuben/manylinux_2_24:2022-03-31-361e6b6 \
  --amend ghcr.io/reuben/manylinux_2_24_x86_64:2022-03-31-361e6b6 \
  --amend ghcr.io/reuben/manylinux_2_24_aarch64:2022-03-31-361e6b6
$ docker manifest push ghcr.io/reuben/manylinux_2_24:2022-03-31-361e6b6

The above lets me write a Dockerfile with FROM ghcr.io/reuben/manylinux_2_24:2022-03-31-361e6b6 and build it for multiple platforms with a single docker buildx build --platforms linux/amd64,linux/arm64 . call.

[reuben]

(This is particularly useful for developing on ARM-based Macs, as ARM Linux containers run way faster than QEMU-emulated x86_64 images on Docker there. But it also lets you simplify creation of Python packages for multiple platforms if your build system is already multi platform capable.)

[mayeut]

Thanks @reuben for this proposal. I'm open to review a PR implementing this. It probably needs to be a post-process step waiting for GHA & Travis-CI steps to be completed as we depend on multiple CI providers to build for different architectures (using QEMU on GHA is a no-go, moving x86_64/i686 to TravisCI is a no-go as well).

[reuben]

FWIW QEMU works for us on GitHub Actions with the buildx builder: https://github.com/docker/setup-buildx-action#with-qemu

[reuben]

But a post-process step that just builds and pushes a multi-platform manifest is probably the simplest implementation.

[mayeut]

FWIW QEMU works for us on GitHub Actions with the buildx builder

Explanation for no-go in the previous comment:

• GHA often times out with 6 hours jobs when building images using QEMU, that's part of why it's a no-go (it was used at some point when having troubles with ppc64 on TravisCI, this was not sustainable).
• We have very low resources available on Travis-CI and that's why moving x86_64/i686 on TravisCI is a no-go.

[messense]

Hi @mayeut, what do you think about the approach in #1412?

[mayeut]

Thanks for looking into this @messense. I'll comment in #1412 directly.

[lukewiwa]

Regarding this I've used the docker manifest command to mirror and upload a multiarch manifest to dockerhub.

Here's the repo that does the action, the bash script feels relatively straight forward. I feel like this might work on the quay platform too.