Buildx not work properly on Mac with Apple Silicon chip

[amizurov]

Contributing guidelines

• [X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• [X] ... the documentation does not mention anything about my problem
• [X] ... there are no open or closed issues that are related to my problem

Description

We use https://github.com/fabric8io/fabric8-maven-plugin to create multi-platform architecture images. And everything works correctly on Docker Desktop with Mac and Intel chip (amd64), but when we run build on Mac with Silicon chip (arm64) we got the error (125):

 docker --config /Users/myuser/projects/container_template/base-image-ubuntu/target/docker-output/docker.artifactory.mycompany.com/containers/base-image-ubuntu/1.1.0/docker buildx create --driver docker-container --name maven
[INFO] DOCKER> unknown flag: --driver
[INFO] DOCKER> See 'docker --help'.
[INFO] DOCKER> 
[INFO] DOCKER> Usage:  docker [OPTIONS] COMMAND
[INFO] DOCKER> 
[INFO] DOCKER> A self-sufficient runtime for containers
...
 DOCKER> Error status (125) while creating builder maven

Also i execute the simple command (docker --config /tmp/docker buildx) and receive difference results. Apple Silicon:

docker --config /tmp/docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'

Apple Intel:

Usage:  docker buildx [OPTIONS] COMMAND

Extended build capabilities with BuildKit

Options:
      --builder string   Override the configured builder instance

Management Commands:
  imagetools  Commands to work on images in registry

Commands:
  bake        Build from a file
  build       Start a build
  create      Create a new builder instance
  du          Disk usage
  inspect     Inspect current builder instance
  ls          List builder instances
  prune       Remove build cache
  rm          Remove a builder instance
  stop        Stop builder instance
  use         Set the current builder instance
  version     Show buildx version information

Run 'docker buildx COMMAND --help' for more information on a command.

Both devices have the same version of Docker Desktop.

Expected behaviour

No exceptions when using buildx, the behavior is the same for both architectures.

Actual behaviour

See description

Buildx version

github.com/docker/buildx v0.10.5 86bdced7766639d56baa4c7c449a4f6468490f87

Docker info

Client:
 Version:    24.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /Users/myuser/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /Users/myuser/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     /Users/myuser/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     /Users/myuser/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-scout

Server:
 Containers: 5
  Running: 0
  Paused: 0
  Stopped: 5
 Images: 716
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.765GiB
 Name: docker-desktop
 ID: 305ec8ab-f713-4696-8916-0a1dabb007da
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE       DRIVER/ENDPOINT             STATUS   BUILDKIT                              PLATFORMS
maven           docker-container
  maven0        unix:///var/run/docker.sock inactive
default         docker
  default       default                     running  v0.11.7-0.20230525183624-798ad6b0ce9f linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
desktop-linux * docker
  desktop-linux desktop-linux               running  v0.11.7-0.20230525183624-798ad6b0ce9f linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Configuration

docker --config /tmp/docker buildx

Build logs

No response

Additional info

No response

[crazy-max]

What's the outcome of the following command?:

$ docker buildx create --driver docker-container --name maven

docker --config /Users/myuser/projects/container_template/base-image-ubuntu/target/docker-output/docker.artifactory.mycompany.com/containers/base-image-ubuntu/1.1.0/docker buildx create --driver docker-container --name maven [INFO] DOCKER> unknown flag: --driver [INFO] DOCKER> See 'docker --help'. [INFO] DOCKER> [INFO] DOCKER> Usage: docker [OPTIONS] COMMAND [INFO] DOCKER> [INFO] DOCKER> A self-sufficient runtime for containers ... DOCKER> Error status (125) while creating builder maven

Looks like this config folder is broken. What's the output of:

docker --config /Users/myuser/projects/container_template/base-image-ubuntu/target/docker-output/docker.artifactory.mycompany.com/containers/base-image-ubuntu/1.1.0/docker info

Also i execute the simple command (docker --config /tmp/docker buildx) and receive difference results. Apple Silicon:

docker --config /tmp/docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'

I'm not able to repro as well:

docker --config /tmp/docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT                              PLATFORMS
default * docker
  default default         running v0.11.7-0.20230525183624-798ad6b0ce9f linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

What's the output of:

docker --config /tmp/docker info

Closing this one as it seems an issue with your environment.

[amizurov]

@crazy-max Hey, Thanks for the reaction of this issue, but i think you cannot reproduce it because you run docker --config /tmp/docker buildx ls on amd64 architecture. I have reproduces this on at least 4 machines (Apple M1 Pro MacOS Ventura 13.5). If you think that the problem with environments i will be really appreciate any help to identify it and please reopen this issue because it not closed, thanks.

[JosephZelada]

I'll echo that I'm experiencing this same issue on an M2 Max Mac on macOS Ventura 13.5. A fix or guidance would be deeply appreciated, as I'm trying to get one of our legacy applications running with this, specifically with the docker-maven-plugin

[crazy-max]

@amizurov No issue for me on M1:

$ docker --config /tmp/docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default * docker                                       
  default default         running v0.11.6+0a15675913b7 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

$ uname -a
Darwin mac-mini-m1.home.foo.com 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103 arm64

What version of Docker Desktop btw?

[nicks]

I believe --config defaults to ~/.docker, so that buildx is loaded from ~/.docker/cli-plugins.

If you set --config=/tmp/docker, it will stop looking in ~/.docker/cli-plugins, and fall back to looking in the system directories for buildx.

From your docker info, I'm guessing you only have buildx installed in ~/.docker/cli-plugins, so setting this flag breaks your env.

[JosephZelada]

Mine is 4.20.1

More output for the curious

$ docker --config /tmp/docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'
$ uname -a
Darwin ML-???? 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:23 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6020 arm64
$ 

[amizurov]

What version of Docker Desktop btw?

Version 4.21.1 (114176)

uname -a
Darwin macBook 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64

[amizurov]

I believe --config defaults to ~/.docker, so that buildx is loaded from ~/.docker/cli-plugins.

If you set --config=/tmp/docker, it will stop looking in ~/.docker/cli-plugins, and fall back to looking in the system directories for buildx.

From your docker info, I'm guessing you only have buildx installed in ~/.docker/cli-plugins, so setting this flag breaks your env.

I have the same setup on my MacBook amd64 and docker --config /tmp/docker buildx ls works properly on it. I'm agreed that something wrong with --config command but i have no idea what exactly and only on aarch64.

[crazy-max]

@amizurov @JosephZelada

What's the output of ls -al /usr/local/lib/docker/cli-plugins/ ?

FWIW we are checking plugins in multiple directories: https://github.com/docker/cli/blob/e0e27724390cbce21cf6d67568972a4227b07382/cli-plugins/manager/manager_unix.go#L5-L8

In my case they are located in both /usr/local/lib/docker/cli-plugins/ and /Users/<name>/.docker/cli-plugins.

[amizurov]

@crazy-max On my amd64 machine ls -al /usr/local/lib/docker/cli-plugins/:

docker-buildx
docker-compose
...

But on aarch64 ls -al /usr/local/lib/docker/cli-plugins/:

No such file or directory

[JosephZelada]

Same "No such file or directory" result running that ls -al on my machine

[crazy-max]

Thanks for your feedback. Reopening and moving this issue to docker/for-mac repo as this is linked to plugin handling on Docker Desktop.

[JosephZelada]

@crazy-max This should have the status/triage label put back on it, shouldn't it?

[drcca]

In case this helps anyone. I ran into this issue with these versions:

Docker version 24.0.5, build ced0996 Docker desktop 4.22.1 (118664)

I then upgraded to the latest and greatest 4.23.0 and 24.0.6 and that still didn't help.

A colleague had a setup that worked but was on an older version. One major difference was that he had this symlink (as alluded to above). /usr/local/lib/docker/cli-plugins -> /Applications/Docker.app/Contents/Resources/cli-plugins Once I created that symlink stuff started to work again. Not sure if this is correct or could end up masking other problems but I decided to put it out there... (I'm sure most people have cli-plugins living under ~/.docker/cli-plugins which symlink to the same directory under /Applications)

[zburgermeiszter]

@drcca you saved my day! Thanks.

[Aeonitis]

I have documented the temporary solution as a gist here, hope the shell script helps others until the issue is resolved.