Colima and docker instability: randomly freezing, requiring reset

[jacksonhull]

Description

I've been trying to troubleshoot an issue with Colima I've been seeing for weeks.

Colima will install and start fine, and I can start my ~20 containers without issue. However, mostly randomly, colima and docker will suddenly be completely unresponsive and containers will stop.

The only remedy is to restart the mac and run colima delete and start over.

I'm running on an Intel i9 with 32GB RAM on MacOS Sonoma 14.4.1 (23E224).

I'm hoping someone has been able to resolve the instability issues with colima running docker on a Mac. The performance of docker on colima is so much better than Docker Desktop and I'd rather not have to go back to it!

I'm happy to further troubleshoot, but hd.stderr.log isn't providing many clues.

Version

colima version 0.6.9 git commit: c3a31ed05f5fab8b2cdbae835198e8fb1717fd0f

runtime: docker arch: x86_64 client: v26.1.3 server: v26.1.1 limactl version 0.22.0 qemu-img version 9.0.0

Operating System

• [ ] macOS Intel <= 13 (Ventura)
• [X] macOS Intel >= 14 (Sonoma)
• [ ] Apple Silicon <= 13 (Ventura)
• [ ] Apple Silicon >= 14 (Sonoma)
• [ ] Linux

Output of colima status

INFO[0000] colima is running using macOS Virtualization.Framework INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/johndoe/.colima/default/docker.sock

Reproduction Steps

1. Reset colima and start using a default.yaml
2. Runs fine for a few days
3. Containers stop and colima and docker freeze, or fail.

Expected behaviour

Run smoothly until manual restart.

Additional context

Default.yaml:

# Number of CPUs to be allocated to the virtual machine.
# Default: 2
cpu: 16

# Size of the disk in GiB to be allocated to the virtual machine.
# NOTE: changing this has no effect after the virtual machine has been created.
# Default: 60
disk: 120

# Size of the memory in GiB to be allocated to the virtual machine.
# Default: 2
memory: 24

# Architecture of the virtual machine (x86_64, aarch64, host).
# Default: host
arch: x86_64

# Container runtime to be used (docker, containerd).
# Default: docker
runtime: docker

# Set custom hostname for the virtual machine.
# Default: colima
#          colima-profile_name for other profiles
hostname: colima

# Kubernetes configuration for the virtual machine.
kubernetes:
  # Enable kubernetes.
  # Default: false
  enabled: false
  
  # Kubernetes version to use.
  # This needs to exactly match a k3s version https://github.com/k3s-io/k3s/releases
  # Default: latest stable release
  version: v1.28.3+k3s2
  
  # Additional args to pass to k3s https://docs.k3s.io/cli/server
  # Default: traefik is disabled
  k3sArgs:
    - --disable=traefik

# Auto-activate on the Host for client access.
# Setting to true does the following on startup
#  - sets as active Docker context (for Docker runtime).
#  - sets as active Kubernetes context (if Kubernetes is enabled).
# Default: true
autoActivate: true

# Network configurations for the virtual machine.
network:
  # Assign reachable IP address to the virtual machine.
  # NOTE: this is currently macOS only and ignored on Linux.
  # Default: false
  address: false
  
  # Custom DNS resolvers for the virtual machine.
  #
  # EXAMPLE
  # dns: [8.8.8.8, 1.1.1.1]
  #
  # Default: []
  dns: []
  
  # DNS hostnames to resolve to custom targets using the internal resolver.
  # This setting has no effect if a custom DNS resolver list is supplied above.
  # It does not configure the /etc/hosts files of any machine or container.
  # The value can be an IP address or another host.
  #
  # EXAMPLE
  # dnsHosts:
  #   example.com: 1.2.3.4
  dnsHosts: {}

# ===================================================================== #
# ADVANCED CONFIGURATION
# ===================================================================== #

# Forward the host's SSH agent to the virtual machine.
# Default: false
forwardAgent: false

# Docker daemon configuration that maps directly to daemon.json.
# https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file.
# NOTE: some settings may affect Colima's ability to start docker. e.g. `hosts`.
#
# EXAMPLE - disable buildkit
# docker:
#   features:
#     buildkit: false
#
# EXAMPLE - add insecure registries
# docker:
#   insecure-registries:
#     - myregistry.com:5000
#     - host.docker.internal:5000
#
# Colima default behaviour: buildkit enabled
# Default: {}
docker: {}

# Virtual Machine type (qemu, vz)
# NOTE: this is macOS 13 only. For Linux and macOS <13.0, qemu is always used.
#
# vz is macOS virtualization framework and requires macOS 13
#
# Default: qemu
vmType: vz

# Utilise rosetta for amd64 emulation (requires m1 mac and vmType `vz`)
# Default: false
rosetta: false

# Volume mount driver for the virtual machine (virtiofs, 9p, sshfs).
#
# virtiofs is limited to macOS and vmType `vz`. It is the fastest of the options.
#
# 9p is the recommended and the most stable option for vmType `qemu`.
#
# sshfs is faster than 9p but the least reliable of the options (when there are lots
# of concurrent reads or writes).
#
# Default: virtiofs (for vz), sshfs (for qemu)
mountType: virtiofs

# Propagate inotify file events to the VM.
# NOTE: this is experimental.
mountInotify: true

# The CPU type for the virtual machine (requires vmType `qemu`).
# Options available for host emulation can be checked with: `qemu-system-$(arch) -cpu help`.
# Instructions are also supported by appending to the cpu type e.g. "qemu64,+ssse3".
# Default: host
cpuType: ""

# Custom provision scripts for the virtual machine.
# Provisioning scripts are executed on startup and therefore needs to be idempotent.# Default: []
provision: []

# Modify ~/.ssh/config automatically to include a SSH config for the virtual machine.
# SSH config will still be generated in ~/.colima/ssh_config regardless.
# Default: true
sshConfig: true

# Configure volume mounts for the virtual machine.
# Colima mounts user's home directory by default to provide a familiar
# user experience.
# Colima default behaviour: $HOME and /tmp/colima are mounted as writable.
# Default: []
mounts:
  - location: /Volumes/Drive
    writable: true
# Environment variables for the virtual machine.
#
# EXAMPLE
# env:
#   KEY: value
#   ANOTHER_KEY: another value
#
# Default: {}
env: {}`

I note the following is almost always present in ha.stderr.log but I can't correlate it directly to the freezes.

{"error":"failed to run [ssh -F /dev/null -o IdentityFile=\"/Users/johndoe/.colima/_lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^[email protected],[email protected]\" -o User=johndoe -o ControlMaster=auto -o ControlPath=\"/Users/johndoe/.colima/_lima/colima/ssh.sock\" -o ControlPersist=yes -T -O forward -L 0.0.0.0:8009:[::]:8009 -N -f -p 49304 127.0.0.1 --]: \"\": exit status 255","level":"warning","msg":"failed to set up forwarding tcp port 8009 (negligible if already forwarded)","time":"2024-05-12T09:55:03+01:00"}

[AndrewGatenbyVS]

I get the same, quite regularly (2-3 times a day, for the last couple of weeks). I don't get anything appended or in the ha.stderr.log file though. Checking all the .log files, there's nothing out of the ordinary and everything suggests that the virtual environment and containers are still running.

However, colima status will hang and not output any response. Similarly, docker ps also does. The only way I've found to address this problem is to do a full reboot of my machine.

Macbook M2 Pro 16", 32GB RAM running MacOS 13.6.2 (22G320)

$ colima version
colima version 0.6.9
git commit: c3a31ed05f5fab8b2cdbae835198e8fb1717fd0f

$ docker -v
Docker version 26.1.4, build 5650f9b102

Colima started, as-per:

colima start --network-address --cpu 8 --memory 8 --disk 60 --dns 1.1.1.1 --dns 8.8.8.8

I can share my colima.yaml file if it's of use, but it's largely the same as @jacksonhull has already provided - bar the options I pass in the colima start command, and my CPU arch is aarch64 and vmType is qemu.

[abiosoft]

Can you try the latest development version of Colima brew install --head colima and see if the issue persists?

Thanks.

[vladttt1123]

I am using the version: colima version 0.8.0 and I jeep getting the exact same problem. is someone working on the resolution of this issue ?

[addy999]

Same here, on latest version :/

[fede]

I've been having the same issue the last couple of days. What I've been able to debug is that:

• Docker daemon inside the vm is still up & running
• Containers are still up & running But from the Mac it's not able to read the docker sock, neither reach the bridged network. It's something in between Colima vm and the host.

If I restart Colima it fixes itself. I've now enabled verbose to see if I can see anything else but I haven't been able so far to find any logs.

Colima version 0.8.1