colima
colima copied to clipboard
abiosoft
Cross-compiling `linux/amd64` on M1 mac
Hi, I'm trying to cross-compile an image using docker buildx and Colima but it seems to just hang on the build step...
How can I debug this? I looked at the logs but didn't see anything suspicious. The qemu VM seems idle....
docker buildx build --platform=linux/amd64 -t studio .
[+] Building 1236.6s (6/12)
=> [internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 103B 0.2s
=> [internal] load .dockerignore 0.2s
=> => transferring context: 106B 0.2s
=> [internal] load metadata for docker.io/bitwalker/alpine-elixir-phoenix:latest 0.4s
=> [internal] load build context 0.0s
=> => transferring context: 33.33kB 0.0s
=> CACHED [1/7] FROM docker.io/bitwalker/alpine-elixir-phoenix:latest@sha256:2c5773461305d76ebb32e23cbd1eeac718f3a1f2435d3b05dbae291f015fc0c3 0.0s
=> CANCELED [2/7] RUN mix do local.hex --force, local.rebar --force 1236.0s
colima version
colima version 0.3.2
git commit: 272db4732b90390232ed9bdba955877f46a50552
runtime: docker
arch: aarch64
client: v20.10.12
server: v20.10.11
I'm on Intel, but a few months ago I was cross-compiling for ARM without issue simply by passing --platform as above. Now, I can't run cross-architecture at all. It looks like over at https://github.com/abiosoft/colima/issues/44#issuecomment-952281801 and https://github.com/abiosoft/colima/releases/tag/v0.3.0 things were changed so that now colima has separate VMs for each architecture.
Is there a reason to have separate VMs? For example, when testing Home Assistant builds, it will build containers all within one command for all of the different architectures.
Nothing changed, the separate VM was just an added option for anyone that prefers that.
@deviantintegral what is the output of docker buildx ls for you?
docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
colima * docker
colima colima running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6
default docker-container
default default Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Looks like buildx is missing entirely. It sounds like buildx is included in deb and rpm packages, but not alpine? I'm also not seeing any qemu packages installed.
https://docs.docker.com/buildx/working-with-buildx/
https://github.com/docker/buildx/issues/305#issuecomment-642709911
docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'
I also used to be able to run cross-platform images, and not just through buildx:
❯ colima version
colima version HEAD-784cec0
git commit: 784cec04dc126b02f5d2697a864282d5d2301afe
runtime: docker
arch: x86_64
client: v20.10.12
server: v20.10.11
❯ colima status
INFO[0000] colima is running
INFO[0000] runtime: docker
INFO[0000] arch: x86_64
❯ docker run -it --rm --platform linux/aarch64 alpine:latest sh
standard_init_linux.go:228: exec user process caused: exec format error
It would be nice if colima could alert when there's a new VM image available. Should I file an issue for that?
Given how it seems you expected buildx to be present, I deleted the VM and recreated it. A new version was downloaded, and I'm able to run sh with aarch64. However, docker buildx ls still fails?
❯ docker run -it --rm --platform linux/aarch64 alpine:latest uname -a
Linux 0e20f7b7b87e 5.10.88-0-virt #1-Alpine SMP Tue, 28 Dec 2021 12:51:14 +0000 aarch64 Linux
❯ docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'
buildx isn't available inside the VM either:
❯ colima ssh
colima:/Users/andrew$ docker buildx
docker: 'buildx' is not a docker command.
See 'docker --help'
colima:/Users/andrew$
It does look like docker build --platform=linux/arm64 works as expected.
Looks like
buildxis missing entirely. It sounds like buildx is included in deb and rpm packages, but not alpine? I'm also not seeing any qemu packages installed.
It is not missing, it actually needs to be installed on same host as the docker client.
Hi, I'm trying to cross-compile an image using docker buildx and Colima but it seems to just hang on the build step...
How can I debug this? I looked at the logs but didn't see anything suspicious. The qemu VM seems idle....
@lpgauth is the Dockerfile something you can share? I can try building it on my end as well.
@abiosoft
git clone https://github.com/lpgauth/live_upload_example.git
cd live_upload_example
docker buildx build --platform=linux/amd64 -t test .
@lpgauth @abiosoft
I've made some discoveries, when I adjust the ulimit it seems to sometimes succeed and sometimes fail; on failure a core dump file is generated; I'm not familiar with erlang so I can't provide more information.
How to modify ulimit:
# install vim
colima ssh
sudo apk add vim
# change 'rc_ulimit' line to 'rc_ulimit="-c unlimited -n 65536 -i 65536 -s 65536 -l 65536 -u unlimited"'
vim /etc/init.d/docker
vim /etc/init.d/buildkitd
# restart colima
colima stop && colima start
@mritd interesting, I will try to replicate and have a look at the core dump.
Same problem. Frequent hangs during build or run with --platform linux/amd64.
-> % colima version
colima version 0.5.4
git commit: feef4176f56a7dea487d43689317a9d7fe9de27e
runtime: docker
arch: aarch64
client: v23.0.3
server: v20.10.20
-> % colima status
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/xxx/.colima/default/docker.sock
-> % docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
colima * docker
colima colima running 20.10.20 linux/arm64, linux/amd64
default error
desktop-linux error
Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot load builder desktop-linux: Cannot connect to the Docker daemon at unix:///Users/xxx/.docker/run/docker.sock. Is the docker daemon running?
Is there anything I can do?
docker buildx ls NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS colima * docker colima colima running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6 default docker-container default default Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
@lpgauth Did you have to do something special to get all these different emulated architectures for colima? I am running on a Mac M1 Max, is there a limitation or workaround?
Same issue when compiling golang, I tried adjusting the ulimit but it didn't seem to make any difference.
I think I have a similar problem. While I currently do not mind not being able to build for linux/amd64, it sure would be nice to be able to.
I never had Docker Desktop on this MacBook (M1, MacOS 13.5). I installed colima, docker, docker-compose and docker-buildx. The hello-world container works.
This is what docker buildx ls returns:
xx@~/some-project (master) docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
colima * docker
colima colima running 23.0.6 linux/arm64
default error
desktop-linux error
Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cannot load builder desktop-linux: Cannot connect to the Docker daemon at unix:///Users/igvo/.docker/run/docker.sock. Is the docker daemon running?
I am somehow missing linux/amd64 altogether.
This is my colima.yaml (I removed the comments for brevity):
cpu: 2
disk: 60
memory: 2
arch: aarch64
runtime: docker
kubernetes:
enabled: false
version: v1.27.1+k3s1
k3sArgs:
- --disable=traefik
autoActivate: true
network:
address: false
dns: []
dnsHosts: {}
driver: gvproxy
forwardAgent: false
docker: {}
vmType: vz
rosetta: true
mountType: virtiofs
mountInotify: true
cpuType: ""
layer: false
provision: []
sshConfig: true
mounts: []
env: {}
Can anyone see any reason why linux/amd64 is not listed in docker buildx ls?
~~And does anyone know how to get rid of the default and desktop-linux lines - since they apparently are not available?~~ Resolved it by creating a symlink for default (sudo ln /Users/xx/.colima/default/docker.sock /var/run/docker.sock) and removing desktop-linux (docker context rm desktop-linux).
@luisdavim
starting
colimawith--vm-type vz --vz-rosettadoes seem to help.
I'm using the following to start colima...how would I incorporate the above?
brew services start colima
Information:
$ colima version
colima version 0.6.3
git commit: a9df8ba55e2ec9c5a9ecc4fb2ab941634b58ed3b
runtime: docker
arch: aarch64
client: v24.0.7
server: v24.0.5
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
colima docker
colima colima running v0.11.6+0a15675913b7 linux/arm64, linux/amd64, linux/amd64/v2
I am looking to use linux/s390x as an emulator.
Note: When doing a colima stop and colima start --vm-type vz --vz-rosetta I get the following:
FATA[0000] error starting vm: error at 'starting': exit status 1
Any hints or ideas anyone?