fondant
fondant copied to clipboard
Make pulling images on M1 / ARM silicon more user friendly
Currently users who try to pull images on their M1 silicon (eg. using the local runner) will run into the following issue:
no matching manifest for linux/arm64/v8 in the manifest list entries
They can solve this by setting the DOCKER_DEFAULT_PLATFORM
environment variable.
export DOCKER_DEFAULT_PLATFORM=linux/amd64
This will instruct docker to pull images built for the linux/amd64
platform instead. From the docker docs:
Some images do not support the ARM64 architecture. You can add --platform linux/amd64 to run (or build) an Intel image using emulation.
However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.
In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. This issue should become less common over time, as more and more images are rebuilt supporting multiple architectures.
I see different actions we can take with different levels of investment and return:
- Document the current workaround clearly
- Add the
platform
flag / environment variable in the docker compose file when compiling the pipeline - Build arm64 or multi-arch images. We should investigate what is needed for this and how it will affect our build times, which are already quite long building for a single architecture.
### Tasks
- [ ] https://github.com/ml6team/fondant/issues/690