manylinux
manylinux copied to clipboard
Publish multi-platform images
It'd be great if the project could publish multi-platform images so that one can do multi-arch Docker builds using buildx and e.g. docker buildx build --platforms linux/amd64,linux/arm64
.
See some relevant docs:
- https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/
- https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/
Right now, because the images for different architectures have different names, one either has to create a manifest by hand, or use separate downstream Dockerfiles for each arch. I've tried out with my own manifest and it works well for multi-platform builds:
$ docker pull quay.io/pypa/manylinux_2_24_aarch64:2022-03-31-361e6b6
$ docker pull quay.io/pypa/manylinux_2_24_x86_64:2022-03-31-361e6b6
$ docker tag quay.io/pypa/manylinux_2_24_x86_64:2022-03-31-361e6b6 ghcr.io/reuben/manylinux_2_24_x86_64:2022-03-31-361e6b6
$ docker tag quay.io/pypa/manylinux_2_24_aarch64:2022-03-31-361e6b6 ghcr.io/reuben/manylinux_2_24_aarch64:2022-03-31-361e6b6
$ docker login ghcr.io
$ docker push ghcr.io/reuben/manylinux_2_24_x86_64:2022-03-31-361e6b6
$ docker push ghcr.io/reuben/manylinux_2_24_aarch64:2022-03-31-361e6b6
$ docker manifest create ghcr.io/reuben/manylinux_2_24:2022-03-31-361e6b6 \
--amend ghcr.io/reuben/manylinux_2_24_x86_64:2022-03-31-361e6b6 \
--amend ghcr.io/reuben/manylinux_2_24_aarch64:2022-03-31-361e6b6
$ docker manifest push ghcr.io/reuben/manylinux_2_24:2022-03-31-361e6b6
The above lets me write a Dockerfile with FROM ghcr.io/reuben/manylinux_2_24:2022-03-31-361e6b6
and build it for multiple platforms with a single docker buildx build --platforms linux/amd64,linux/arm64 .
call.
(This is particularly useful for developing on ARM-based Macs, as ARM Linux containers run way faster than QEMU-emulated x86_64 images on Docker there. But it also lets you simplify creation of Python packages for multiple platforms if your build system is already multi platform capable.)
Thanks @reuben for this proposal. I'm open to review a PR implementing this. It probably needs to be a post-process step waiting for GHA & Travis-CI steps to be completed as we depend on multiple CI providers to build for different architectures (using QEMU on GHA is a no-go, moving x86_64/i686 to TravisCI is a no-go as well).
FWIW QEMU works for us on GitHub Actions with the buildx builder: https://github.com/docker/setup-buildx-action#with-qemu
But a post-process step that just builds and pushes a multi-platform manifest is probably the simplest implementation.
FWIW QEMU works for us on GitHub Actions with the buildx builder
Explanation for no-go in the previous comment:
- GHA often times out with 6 hours jobs when building images using QEMU, that's part of why it's a no-go (it was used at some point when having troubles with ppc64 on TravisCI, this was not sustainable).
- We have very low resources available on Travis-CI and that's why moving x86_64/i686 on TravisCI is a no-go.
Hi @mayeut, what do you think about the approach in #1412?
Thanks for looking into this @messense. I'll comment in #1412 directly.
Regarding this I've used the docker manifest command to mirror and upload a multiarch manifest to dockerhub.
Here's the repo that does the action, the bash script feels relatively straight forward. I feel like this might work on the quay platform too.