WIP: Support arm64 architecture to enable usage on the current M1 MacBooks

[bweinzierl]

trafficstars

This PR shows an example how to generate the dockerfiles to support arm architecture. Also it tries to give a proposal how to modify the jinja files to get the desired results. This part is actually not tested.

Modification of dockerfile files I modified the Dockerfile docker/php-official/7.4/Dockerfile directly to show the desired result when they are automatically generated for this example. This dockerfile can be bild on ARM (tested on the latest MacBook Pro) and also on a MacBook with Intel Chip (Tested on MacBook Pro (16-inch, 2019)).

Modification of jinja2 files Without knowing if the syntax is correct i also modified the jinja templates. With the modifications the dockerfiles should be generated in the shown way.

I removed the argument arch in the macro 'gosu(path="/sbin", arch="amd64", version="1.10")' as it is determined dynamicly during build. I did not find any usage of the argument. Hope that is ok...

[htuscher]

This will take some time to review as I have to check if I need to do docker buildx with different architectures on different hardware in CI. Have no idea about building vor arm64 yet

[bweinzierl]

As far as i understand it it is doable on just one architecture. There is a --platform argument that looks something like that: docker buildx build --platform linux/amd64,linux/arm64 . But there are indeed some prerequisites. It does not work out of the box on my mac. Docker needs to be configured in some way. I get the error: error: multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")

Maybe i can help by setting up an example...

[htuscher]

Yes an example would be great.

[bweinzierl]

I found this example online (.gitlab-ci.yml) ... but not tested it yet. As it looks it really should only be a super simple thing. You need a docker image with buildx installed (i did not look at the proposed image, maybe it makes sense to use something official or install buildx manually) and run these two commands...

image: jdrouet/docker-with-buildx:stable

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2

services:
  - docker:dind

build:
  stage: build
  script:
    - docker login -u "$DOCKER_USERNAME" -p "$DOCKER_PASSWORD"
    - docker buildx create --use
    - docker buildx build --push --platform linux/amd64,linux/arm64 --tag your-username/multiarch-example:gitlab .

[bweinzierl]

Thanx to @allansun for PHP 8.0 example

[allansun]

Thanks to @bweinzierl I've managed to get webdevops/php-nginx-dev:8.0-alpine compiled under arm64 on my M1 MacBookPro.

I DID NOT use the 'docker-container' driver ('docker buildx create --use'), instead I used the default 'docker' driver ('docker buildx use default'), because this is the only way I can get buildx to use local built images. ( toolbox -> php -> php-nginx -> php-nginx-dev)

To get @hhoechtl running this in CI environment, you DO need to use the 'docker-container' driver ('docker buildx create --use'). What it does is to bring up a docker-in-docker builder, which supports multi arch build including arm64. Even though I couldn't try it myself (because I cannot push to webdevops/* to get later images using correct inheritance), the method @bweinzierl mentioned seems to be good enough. We just need to alter the python code to use 'docker buildx build' instead of 'docker build'.

Can't wait to see this happening!

[bweinzierl]

Today i spent quite some time setting up a build task for building with buildx on our gitlab runner. As easy as it looked on first glance, when using buildx on a amd64 machine with qemu there were a few problems.

Getting the runner working

As always it is not as simple as promoted in all the howtos. But i have it working now for our gitlab runner:

image: jdrouet/docker-with-buildx:latest

variables:
  DOCKER_DRIVER: overlay2

services:
  - docker:dind

build:
  stage: build
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
    - apk add --update git
    - git clone --branch testing-buildx https://github.com/Format-D/Dockerfile.git webdevops-fork
    - docker buildx version
    - docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-* # this was needed / without it somehow the arm support is missing
    - docker run --privileged --rm tonistiigi/binfmt --install all
    - docker context create multiarch-builder-context
    - docker context ls
    - docker buildx create --name multiarch-builder --use multiarch-builder-context --platform linux/amd64,linux/arm64
    - docker buildx inspect --bootstrap
    - docker buildx ls
    - docker buildx build --push --platform linux/amd64,linux/arm64 --tag ${CI_REGISTRY_IMAGE}/toolbox:latest webdevops-fork/docker/toolbox/latest
    - docker buildx build --push --platform linux/amd64,linux/arm64 --tag ${CI_REGISTRY_IMAGE}/php:7.4 webdevops-fork/docker/php-official/7.4
#  when: manual
  allow_failure: false

The webdevops golang binaries (gosu and go-replace) throw an error when executed

#8 5.164 runtime: failed to create new OS thread (have 2 already; errno=22) #8 5.164 fatal error: newosproc

Looks like it is some problem with qemu and golang? I just commented the lines were the scripts are executed. This way i successfully built the docker/toolbox image at least.

@hhoechtl Do you happen to know why the golang scripts cannot be called in a qemu environment but native it works?

Update: I found this: https://issueexplorer.com/issue/tianon/gosu/93 Would it be possible to provide new versions of gosu and go-replace?

[allansun]

@bweinzierl from what I can tell it's nothing to do with qemu.

The main problem is that as long as you use 'buildx' to build and in Dockerfile you are 'FROM webdevops/*', this may create problem, because no matter how you tweak the settings, if the 'parent' image was not built for your target arch, you will definitely get some problems later on.

I saw exactly the same link https://issueexplorer.com/issue/tianon/gosu/93 maybe we can give it a try.

[bweinzierl]

The main problem is that as long as you use 'buildx' to build and in Dockerfile you are 'FROM webdevops/*', this may create problem, because no matter how you tweak the settings, if the 'parent' image was not built for your target arch, you will definitely get some problems later on.

That is not the problem. I am building the whole chain. Starting with toolbox:latest which depends on alpine:latest. So this should work (and does in deed work on my M1). But on amd with buildx the call of /usr/local/bin/go-replace" --version in toolbox throws this error (it is the correct binary for arm downloaded). The link i found describes a similar problem with go and qemu.

[allansun]

That is not the problem. I am building the whole chain. Starting with toolbox:latest which depends on alpine:latest. So this should work (and does in deed work on my M1). But on amd with buildx the call of /usr/local/bin/go-replace" --version in toolbox throws this error (it is the correct binary for arm downloaded). The link i found describes a similar problem with go and qemu.

When you build locally with buildx, the webdevops/toolbox:latest is built successfully and it's available in you 'docker images', you would think buildx would use such local image when you start building webdevops/php:7.4-alpine.

However according to my test (and some google results, which I can't remember where...), buildx will ignore your local image and still pull remote images.

Unless buildx is able to use local image (not possible at the moment), the only way we can get it working is to push the built image to docker hub...

[bweinzierl]

@allansun The toolbox build is already failing so the chain is not the problem. It is build on alpine:latest and this is a multiarch image. I would have internal Gitlab Registry to push to... so for testing i could make it work. Without the go update my only option is to build on two different machines and then combine the images to a multiarch with docker manifest commands. I guess this would be tedious but the only option without working go binaries. Do you know go? Maybe we need to fork the gosu and go_replace and update it ourselves, but i have never done anything with go.

[allansun]

@bweinzierl

Check out this: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images and this https://docs.docker.com/engine/reference/builder/#from

Finally, depending on your project, the language that you use may have good support for cross-compilation. In that case, multi-stage builds in Dockerfiles can be effectively used to build binaries for the platform specified with --platform using the native architecture of the build node. A list of build arguments like BUILDPLATFORM and TARGETPLATFORM is available automatically inside your Dockerfile and can be leveraged by the processes running as part of your build.

The reason your build fails should because the alpine:latest wasn't pulled with correct arch in the first place...

[bweinzierl]

@allansun Ah... thanks! This might really be the problem. I will check it out later!

[bweinzierl]

@allansun @hhoechtl I now have a running setup to build the webdevops images as multiarch images and push to our internal gitlab docker registry. The following workaround for the qemu - go - incompatibility was necessary: I am installing the binaries for go-replace and gosu matching to the BUILDPLATFORM which is amd64 in my case and not the right executable for TARGETPLATFORM. That way i prevent qemu from beeing triggered when go-replace is executed during build. This gives me a multiarch image in the end where the two go binaries are for the wrong architecture if you want to run it on an arm processor. So what i am doing is putting a minimal dockerfile on my M1 to pull the image and then replace the go-executables.

So to provide a running setup that is consistent and logical we need to solve the go - qemu problem (by updating).

If we cannot do that then we need to build the chain with amd executables and replace the executables for the correct architecture ones at the end of the build (and don't execute if afterwards). This would result in the same image but is a hacky / ugly solution.

@hhoechtl Could you give an estimation if/when you would be able to provide these new executables or if not would be willing to implement the hacky build pipeline into your current setup. I guess for this i would remove the changes for the architecture specific go-binaries again in this PR and revert to what is there already and only add a last line in the last image in the chain where they are replaced by the correct one.

[htuscher]

Maybe the weekend 5./6. of February. That's the next spare time I have.

[sonarqubecloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[bweinzierl]

We are now using this modified branch with the qemu workaround to build the php-apache-74 and php-apache-80 images as multiarch: https://github.com/Format-D/Dockerfile/tree/temp-qemu-workaround (to make the 8.0 build work i had to remove the install of the sockets php extension in addition to the go workaround)

Here i found additional info about the qemu go problem: https://github.com/moby/qemu/issues/17

Especially the linked comment is interesting, there seems to be some configuration variable to prevent this issue.

@hhoechtl If you find time on this weekend i would be open for a hangout session. Could also help with stuff on that weekend.

[smxsm]

@bweinzierl builds and runs on M1, thank you very much for your efforts! I am trying to build "php-apache:7.4" and "php-apache-dev:7.4" etc., too, but buildx always uses the official webdevops.io base images from the Docker Registry ... looks like buildx does not use/support locally built base images, but always pulls from the registry (see https://github.com/docker/buildx/issues/301) ... So I still end up with AMD base images running qemu processes inside :( Looks like I have to use a custom registry, too, and push the intermediate php images there. I hope this PR lands in the official webdevops images soon!

For reference, here is the script I used to test it locally, without Gitlab CI:

#!/bin/bash
set -e

echo "Building Multiarch Webdevops images ..."

if ! [[ -d ./webdevops-fork ]]; then
    git clone --branch temp-qemu-workaround https://github.com/Format-D/Dockerfile.git webdevops-fork
else
    cd webdevops-fork && git pull && cd ..
fi
docker buildx version
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-* # this was needed / without it somehow the arm support is missing
docker run --privileged --rm tonistiigi/binfmt --install all
if [[ -z `docker context ls | grep multiarch-builder-context` ]]; then
    docker context create multiarch-builder-context
    docker buildx create --name multiarch-builder --use multiarch-builder-context --platform linux/amd64,linux/arm64
fi
docker context ls
docker buildx inspect --bootstrap
docker buildx ls

# to push to a gitlab registry, use e.g.
#docker buildx build --push --platform linux/amd64,linux/arm64 --tag my-webdevops-registry.com/toolbox:latest webdevops-fork/docker/toolbox/latest
#docker buildx build --push --platform linux/amd64,linux/arm64 --tag my-webdevops-registry.com/php:7.4 webdevops-fork/docker/php-official/7.4
#docker buildx build --push --platform linux/amd64,linux/arm64 --tag my-webdevops-registry.com/php:8.0 webdevops-fork/docker/php-official/8.0

# you can only --load one target platform image locally, see https://github.com/docker/buildx/issues/59
docker buildx build --load --platform linux/arm64 --tag webdevops/toolbox:latest webdevops-fork/docker/toolbox/latest
docker buildx build --load --platform linux/arm64 --tag webdevops/php:7.4 webdevops-fork/docker/php-official/7.4
# the following images will use the official webdevops.io images, not our local base image - those are still AMD only :(
# see https://github.com/docker/buildx/issues/301
docker buildx build --load --platform linux/arm64 --tag webdevops/php-apache:7.4 webdevops-fork/docker/php-apache/7.4
docker buildx build --load --platform linux/arm64 --tag webdevops/php-apache-dev:7.4 webdevops-fork/docker/php-apache-dev/7.4
# further php version ...
#docker buildx build --load --platform linux/arm64 --tag webdevops/php-apache-dev:8.0 webdevops-fork/docker/php-apache-dev/8.0

echo "Done"

[bweinzierl]

We are building the complete image chain and pushing them to our internal gitlab registry. So this was no problem for me. I will see if i can provide the images on docker hub too until it is included in webdevops build.

[smxsm]

That would be awesome! Did you need to setup anything so that buildx uses your own base images from your custom registry and not the official webdevops (AMD) images from Github?

[bweinzierl]

@smxsm apache php 7.4 is now available here (and php 8.0 is currently being built): https://hub.docker.com/r/formatdgmbh/webdevops-php-apache-dev/tags https://hub.docker.com/r/formatdgmbh/webdevops-php-apache/tags

Please be aware that this is only intended to be a temporary location until webdevops implements the buildx in their pipeleine.

This is our CI Pipeline. We are using the fork and replacing the FROM ... with our Images (see line 8):

build_webdevops_chain_74:
  stage: build
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
    - docker login -u ******* -p *******
    - apk add --update git
    - git clone --branch temp-qemu-workaround https://github.com/Format-D/Dockerfile.git webdevops-fork
    - find ./webdevops-fork -type f -name "Dockerfile" -print0 | xargs -0 sed -i 's|FROM webdevops/|FROM registry.gitlab.com/format-d/devops/format-d-base-images/|g' || echo "error"
    - cat webdevops-fork/docker/php/7.4/Dockerfile
    - docker buildx version
    - docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-* # this was needed / without it somehow the arm support is missing
    - docker run --privileged --rm tonistiigi/binfmt --install all
    - docker context create multiarch-builder-context
    - docker context ls
    - docker buildx create --name multiarch-builder --use multiarch-builder-context --platform linux/amd64,linux/arm64
    - docker buildx inspect --bootstrap
    - docker buildx ls
    - docker buildx build --push --platform linux/amd64,linux/arm64 --tag ${CI_REGISTRY_IMAGE}/php:7.4 webdevops-fork/docker/php/7.4
    - docker buildx build --push --platform linux/amd64,linux/arm64 --tag ${CI_REGISTRY_IMAGE}/php-apache:7.4 --tag formatdgmbh/webdevops-php-apache:7.4 webdevops-fork/docker/php-apache/7.4
    - docker buildx build --push --platform linux/amd64,linux/arm64 --tag ${CI_REGISTRY_IMAGE}/php-apache-dev:7.4 --tag formatdgmbh/webdevops-php-apache-dev:7.4 webdevops-fork/docker/php-apache-dev/7.4
  when: manual
  allow_failure: false

[smxsm]

@bweinzierl thanks a lot! I did a quick test run with "formatdgmbh/webdevops-php-apache-dev:7.4" ... the weird thing is, I still have "qemu" running inside the container ... can you check if you have that, too? If you're building the whole chain that shouldn't be the case ... strange.

root@2f92565e40ed:/app# ps -aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.5  0.1 175636 42864 ?        Ssl  20:55   0:01 /usr/bin/qemu-x86_64 /usr/bin/python2 /usr/bin/supervisord -c /opt/docker/etc/supervisor.conf --logfile /dev/null --pidf
root       232  0.0  0.1 315800 27712 ?        Sl   20:55   0:00 /usr/bin/qemu-x86_64 /usr/sbin/syslog-ng -F --no-caps -p /var/run/syslog-ng.pid
root       234  0.1  0.3 929376 83684 ?        Sl   20:55   0:00 /usr/bin/qemu-x86_64 /usr/local/bin/php-fpm --nodaemonize
root       235  0.1  0.1 160960 26484 ?        Sl   20:55   0:00 /usr/bin/qemu-x86_64 /usr/sbin/apache2 -DFOREGROUND -DAPACHE_LOCK_DIR
root       238  0.0  0.0 151712 10272 ?        Sl   20:55   0:00 /usr/bin/qemu-x86_64 /usr/sbin/cron -f
root       239  0.0  0.0 150492 12556 ?        Sl   20:55   0:00 /usr/bin/qemu-x86_64 /bin/bash /opt/docker/bin/service.d/postfix.sh
root       240  0.0  0.0 161704 19192 ?        Sl   20:55   0:00 /usr/bin/qemu-x86_64 /usr/sbin/sshd -D
www-data   345  0.0  0.2 2503336 54008 ?       Sl   20:55   0:00 /usr/bin/qemu-x86_64 /usr/sbin/apache2 -DFOREGROUND -DAPACHE_LOCK_DIR
www-data   347  0.0  0.2 2503452 54292 ?       Sl   20:55   0:00 /usr/bin/qemu-x86_64 /usr/sbin/apache2 -DFOREGROUND -DAPACHE_LOCK_DIR
www-data   477  2.6  0.5 962060 123204 ?       Sl   20:55   0:08 /usr/bin/qemu-x86_64 /usr/local/bin/php-fpm --nodaemonize
www-data   479  2.4  0.5 1037472 130256 ?      Sl   20:55   0:07 /usr/bin/qemu-x86_64 /usr/local/bin/php-fpm --nodaemonize
root       763  0.0  0.0 188792 14208 ?        Ssl  20:55   0:00 /usr/bin/qemu-x86_64 /usr/lib/postfix/sbin/master -w
postfix    765  0.0  0.0 188444 15396 ?        Sl   20:55   0:00 /usr/bin/qemu-x86_64 /usr/lib/postfix/sbin/pickup -l -t unix -u -c
postfix    767  0.0  0.0 188532 15524 ?        Sl   20:55   0:00 /usr/bin/qemu-x86_64 /usr/lib/postfix/sbin/qmgr -l -t unix -u
root      1272  2.0  0.0 150744 14116 pts/0    Ssl  21:00   0:00 /usr/bin/qemu-x86_64 /bin/bash
root      1285  2.0  0.0 148168  6748 ?        Sl   21:00   0:00 /usr/bin/qemu-x86_64 /bin/sleep 5
root      1288  0.0  0.0 155868  9636 ?        Rl+  19:12   0:00 /bin/ps -aux

If I run my custom-built single PHP 7.4. image (not using webdevops, but "FROM arm64v8/php:7.4-apache", I get this:

root@485d3b7a84b8:/app# ps -aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.1  87932 27536 ?        Ss   21:03   0:00 apache2 -DFOREGROUND
www-data    19  0.0  0.0  87964  9520 ?        S    21:03   0:00 apache2 -DFOREGROUND
www-data    20  0.0  0.0  87964  9520 ?        S    21:03   0:00 apache2 -DFOREGROUND
www-data    21  0.0  0.0  87964  9520 ?        S    21:03   0:00 apache2 -DFOREGROUND
www-data    22  0.0  0.0  87964  9520 ?        S    21:03   0:00 apache2 -DFOREGROUND
www-data    23  0.0  0.0  87964  9520 ?        S    21:03   0:00 apache2 -DFOREGROUND
root        24  0.0  0.0   3960  3004 pts/0    Ss   21:04   0:00 /bin/bash
root        31  0.0  0.0   6448  2400 pts/0    R+   21:04   0:00 ps -aux

[bweinzierl]

Hey @smxsm, ich checked and i don't have any qemu processes running in my container. Are you running on M1 MacBookPro? What does the docker desktop UI show? It should display a label if the containers are run with emulation.

[smxsm]

Hm, that's really weird ... are you using the same image, "formatdgmbh/webdevops-php-apache-dev:7.4"? Yes, sure, I am on M1 Macbook Pro. About Docker Desktop... not neccessarily, if the top image has "platform" set to ARM64, there is no emulation label, even if all underlying images are built for AMD ... I've made some experiments with that, and qemu only goes away if every single underlying base image is built for ARM (which makes sense of course), but the Desktop UI doesn't really check that properly it seems ...

I thought maybe the base "toolbox" image has to switch between FROM arm64v8/alpine:latest and FROM alpine:latest depending on the build target ... then all images on top of that should be built for ARM?

So, in "ps" I see qemu for all processes, but Docker Desktop doesn't show the label for the Webdevops container:

So it seems that my container is not a "real" ARM container.

[bweinzierl]

Hmm... this is really strange. I always build the last image on my laptop and was always getting a label in docker desktop if one of the images in the chain was not arm. (I tested the formatdgmbh/webdevops-php-apache-dev:7.4 directly without building locally from it of cause - but no qemu processes inside container either). I digged into it some more and there is a qemu-system-aarch64 process directly on my mac that is spiking CPU when i reload the website in my container. So this definitely seems like there is something wrong :(

As it looks even in an arm image there can exist amd binaries that are executed with qemu automatically: https://stackoverflow.com/questions/66834864/how-to-determine-when-docker-containers-on-an-m1-macbook-are-running-via-qemu

[bweinzierl]

OK, now i am totally lost... I ran the following images one after another: docker run -it --entrypoint /bin/sh registry.gitlab.com/format-d/devops/format-d-base-images/toolbox docker run -it --entrypoint /bin/sh alpine:latest docker run -it --entrypoint /bin/sh arm64v8/alpine:latest

In each container i ran this command to create cpu load: cat /dev/zero > /dev/null

In every case the process qemu_system_aarch64 on my mac spikes. So this seems to be normal?

[smxsm]

Hm, I wouldn't expect that qemu_system_aarch64 would spike for the arm64v8 alpine at least ... strange and confusing :( Gotta test that later ... Can you run a "ps" inside the containers what the processes inside look like? For me at least the "ps" output was a good indicator so far ... at least that's what I found when I wrote this blog post: https://www.proudcommerce.com/devops/docker-performance-on-apple-macbook-pro-with-m1-max-processor-status-and-tips

[bweinzierl]

@smxsm Did some more Tests. From inside the container everything seems just fine...

With our image i get no qemu process:

# docker run -it --entrypoint /bin/sh formatdgmbh/webdevops-php-apache-dev:7.4

# ps axu
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.3  0.0   2040   432 pts/0    Ss   09:15   0:00 /bin/sh
root         8  0.0  0.0   7132  2528 pts/0    R+   09:15   0:00 ps axu

With the original devops amd64 image i get a qemu process:

# docker run -it --entrypoint /bin/sh webdevops/php-apache-dev:7.4            

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
# ps axu
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.6  0.0 146504  8132 pts/0    Ssl  09:16   0:00 /usr/bin/qemu-x86_64 /bin/sh
root         8  0.0  0.0 155836  9564 ?        Rl+  Feb02   0:00 /bin/ps axu

Still... i am not satisfied with the performance of a test-project in comparison to my old Macbook. So something is off. But could be file io. By the way, i checked your Blog post... interesting read! We are still using docker-sync but the new VirtioFS sounds promising.

[bweinzierl]

@smxsm Found my performance problem. Has nothing to do with qemu. Was a DNS resolution problem between docker containers. A massive bug on M1 described here: https://github.com/docker/for-mac/issues/5626

The described workaround worked for me. Not i have response times of < 500ms for complex pages (with lots of elasticsearch calls). Before it was 15s -50s.

A whole lot of obstacles to get the M1 running smoothly...

[smxsm]

@bweinzierl wow, good catch 👍🏼 gotta check that, too. Yes, M1 is a bit of a hassle so far 😸 But what's still weird is that I have everything running via qemu when using your image, whereas you don't ... is it possible that you are using a local version of "formatdgmbh/webdevops-php-apache-dev:7.4" what maybe is different to the one from the Docker registry? Or do you see it being pulled from the remote registry? Maybe I should really try to build it myself using a local Docker Registry container to see if that's any different ... we can't really live with qemu since i/o operations inside the container are awefully slow then.