Failure on `colima start` when using docker as runtime

[thegeorgeous]

Description

Running colima start on a fresh install fails with the following log:

INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0000] preparing network ...                         context=vm
INFO[0000] creating and starting ...                     context=vm
INFO[0023] provisioning ...                              context=docker
FATA[0024] error provisioning docker: signal: killed

Version

Colima Version: 0.4.4 Lima Version: 0.11.2 Qemu Version: 7.0.0

Operating System

• [ ] macOS Intel
• [X] macOS M1
• [ ] Linux

Reproduction Steps

1. Install colima
2. Run colima start

Expected behaviour

colima should start without errors

Additional context

colima starts correctly when using the containerd runtime and works as expected. It seems to be an issue only with the docker runtime

From serial.log it seems to have stopped at

Welcome to Alpine Linux 3.14
^MKernel 5.10.109-0-virt on an aarch64 (/dev/ttyAMA0)

^Mcolima login:

[abiosoft]

How did you install colima and docker client, is it via brew?

[thegeorgeous]

@abiosoft yes

brew install colima docker

[abiosoft]

Looks like a weird behaviour I have experienced on macOS Monterey where some binaries get killed for whatever reason.

Though I am yet to experience it with a brew installed binary, mostly on locally generated ones.

Does removing docker and reinstalling fix anything?

[thegeorgeous]

@abiosoft I have already tried reinstalling docker, lima and colima. I've also tried deleting all the configurations for all three before reinstalling. It didn't change anything

[abiosoft]

Can you try the development version?

brew install --HEAD colima

[thegeorgeous]

@abiosoft same results.

colima list returns this output

PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK     RUNTIME    ADDRESS
default    Running    aarch64    2       2GiB      60GiB    docker

Although this was the case for 0.4.4 too.

[abiosoft]

Can you start with colima start --very-verbose and share the full output?

Thanks.

[thegeorgeous]

@abiosoft

This is the output of colima start --very-verbose

TRAC[0000] cmd ["limactl" "info"]
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]
INFO[0000] starting colima
INFO[0000] runtime: docker
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]
INFO[0000] preparing network ...                         context=vm
TRAC[0000] cmd ["/opt/homebrew/bin/colima" "daemon" "status" "default"]
TRAC[0000] cmd ["/opt/homebrew/bin/colima" "daemon" "start" "default" "--gvproxy" "--verbose"]
INFO[0000] starting ...                                  context=vm
TRAC[0000] cmd ["limactl" "start" "colima"]
> Using the existing instance "colima"
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/georgethomas/.lima/colima/serial.log")
> SSH Local Port: 58762
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] The essential requirement 1 of 5 is satisfied
> [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
> [hostagent] The essential requirement 2 of 5 is satisfied
> [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
> [hostagent] The essential requirement 3 of 5 is satisfied
> [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf (/etc/fuse3.conf) to contain \"user_allow_other\""
> [hostagent] The essential requirement 4 of 5 is satisfied
> [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running"
> [hostagent] The essential requirement 5 of 5 is satisfied
> [hostagent] Mounting "/Users/georgethomas" on "/Users/georgethomas"
> [hostagent] Mounting "/tmp/colima" on "/tmp/colima"
> [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/georgethomas/.colima/default/docker.sock" (host)
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/georgethomas/.colima/docker.sock" (host)
> [hostagent] The final requirement 1 of 1 is satisfied
> [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/georgethomas/.lima/colima/ga.sock" (host)
> READY. Run `limactl shell colima` to open the shell.
TRAC[0022] cmd ["lima" "sudo" "cat" "/etc/hosts"]
TRAC[0022] cmd ["lima" "sudo" "cat" "/etc/hosts"]
INFO[0022] provisioning ...                              context=docker
TRAC[0022] cmd ["lima" "sudo" "mkdir" "-p" "/etc/docker"]
TRAC[0022] cmd int ["lima" "sudo" "sh" "-c" "cat > /etc/docker/daemon.json"]
TRAC[0022] cmd ["docker" "context" "inspect" "colima"]
TRAC[0022] cmd ["docker" "context" "create" "colima" "--description" "colima" "--docker" "host=unix:///Users/georgethomas/.colima/default/docker.sock"]
FATA[0022] error provisioning docker: signal: killed

[abiosoft]

Looks like it starts up successfuly but had issue when trying to set up docker client context.

Can you try running this directly and see if it crashes?

docker context create colima --description colima --docker "host=unix:///Users/georgethomas/.colima/default/docker.sock"

[thegeorgeous]

@abiosoft looks like that is the command that is getting killed. When I ran it this is the output I got

[1]    43424 killed     docker context create colima --description colima --docker

[abiosoft]

macOS is killing the docker client for whatever reason.

I have experienced that before with other binaries, what seems to trigger it for me is when a running binary is replaced. My solution back then was to first delete the binary before copying the new one over as you can see here.

However, I am not sure how it applies to brew installation. Maybe manually deleting the docker binary and re-installing with brew (with caching disabled).

Do you have Docker for Mac installed?

[thegeorgeous]

@abiosoft no, I don't have Docker for Mac installed, but I do have podman installed.

[thegeorgeous]

@abiosoft I've tried uninstalling everything and removed all caches, but it still has the same problem.

[abiosoft]

@thegeorgeous as a last resort, try brew install --head docker which builds and installs from source.

[thegeorgeous]

@abiosoft that didn't help either. Colima HEAD also seems to have some problem with accepting the answer to this question

are you sure you want to delete colima and all settings? [y/N] y ^M

The enter command simply add ^M

[abiosoft]

This is the strangest issue I have seen in recent times.

I would try to see if I can reproduce any of these behaviours.

[thegeorgeous]

@abiosoft, if it helps, it is not a one-off issue. Some colleagues of mine also faced the same issue, albeit on a M2 Mac

[abiosoft]

@thegeorgeous are you also on an M2 Mac?

[thegeorgeous]

@abiosoft no, I'm on an M1 mac.

[anhoder]

I also encountered this problem on M1 mac

[nomiddlename]

I had this problem, and removing ~/.docker/config.json fixed it.

[thegeorgeous]

@nomiddlename I don't have ~/.docker/config.json. I don't even have the folder ~/.docker. This happens on a fresh install.

[thegeorgeous]

@abiosoft I no longer have this problem after upgrading macOS to Ventura and colima to

colima version 0.4.6
git commit: 10377f3a20c2b0f7196ad5944264b69f048a3d40

runtime: docker
arch: aarch64
client: v20.10.21
server: v20.10.16

[thegeorgeous]

@abiosoft the issue is back after upgrading to Ventura 13.0.1 (22A400) that was released.