Docker engine not starting and sockets missing during diagnostics

[kimdcottrell]

Description

Hey guys. This is a complicated two parter:

1. I'm having trouble starting Docker Engine on the latest version of Docker Desktop, despite all the virtualization stuff seemingly being fine
2. The diagnostic tool is acting up, and I am starting to suspect there is a missing utility in the latest Docker Desktop. The steps below detail how I got the gather utility to work at all.

Reproduce

First, this started off as a simple "perhaps it's an issue with the virtualization." I was seeing "Docker Engine stopped" in Docker Desktop, so I ran this:

kim@kimtalkstech:~/.docker/desktop $ /opt/docker-desktop/bin/com.docker.diagnose check
Starting diagnostics

[PASS] DD0039: are KVM user permissions configured?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[FAIL] DD0017: can a VM be started? vm has not started: failed to open kmsg.log: open /home/kim/.docker/desktop/log/vm/kmsg.log: no such file or directory
[FAIL] DD0016: is the LinuxKit VM running? prereq failed: can a VM be started?

I created that /home/kim/.docker/desktop/log/vm/kmsg.log: no such file or directory file and set it to 775. That changed the error to [FAIL] DD0017: can a VM be started? vm has not started: vm has not started. Not really an improvement.

Checking the virtualization and kvm abilities, we see that it is fine and the permissions are correct:

kim@kimtalkstech:~/.docker/desktop $ lscpu | grep -i virtualization
Virtualization:                  VT-x
kim@kimtalkstech:~/.docker/desktop $ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
kim@kimtalkstech:~/.docker/desktop $ egrep -q 'vmx|svm' /proc/cpuinfo && echo yes || echo no
yes
kim@kimtalkstech:~/.docker/desktop $ lsmod | grep kvm
kvm_intel             499712  2
kvm                  1347584  1 kvm_intel
irqbypass              16384  1 kvm
kim@kimtalkstech:~/.docker/desktop $ ls -al /dev/kvm
crw-rw----+ 1 root kvm 10, 232 Oct 10 01:46 /dev/kvm
kim@kimtalkstech:~/.docker/desktop $ groups kim
kim : kim adm cdrom sudo dip plugdev users kvm lpadmin docker libvirt

So I decided to do some digging into linuxkit. I downloaded the latest here: https://github.com/linuxkit/linuxkit/releases/tag/v1.0.1, made it executable, built the docker image (linuxkit build docker.yml) from here: https://github.com/linuxkit/linuxkit/blob/master/examples/docker.yml, and ran it via qemu and got this, which shows that linuxkit has the ability to start:

$ linuxkit run qemu docker-initrd.img
SeaBIOS (version 1.16.0-debian-1.16.0-5)
Machine UUID 8411bf36-2760-43a6-93cb-f703d6b30a56


iPXE (https://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+3EFCAE30+3EF0AE30 CA00
                                                                               


Booting from Hard Disk...
Boot failed: not a bootable disk

Booting from DVD/CD...
Boot failed: Could not read from CDROM (code 0003)
Booting from ROM...
iPXE (PCI 00:03.0) starting execution...ok
iPXE initialising devices...ok



iPXE 1.21.1+git-20220113.fbbdc3926-0ubuntu1 -- Open Source Network Boot Firmware
 -- https://ipxe.org
Features: DNS HTTP HTTPS iSCSI NFS TFTP VLAN AoE ELF MBOOT PXE bzImage Menu PXEX
T

net0: 42:85:a2:a7:b6:70 using virtio-net on 0000:00:03.0 (Ethernet) [open]
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net0 42:85:a2:a7:b6:70)...... ok
net0: 10.0.2.15/255.255.255.0 gw 10.0.2.2
net0: fec0::4085:a2ff:fea7:b670/64 gw fe80::2
net0: fe80::4085:a2ff:fea7:b670/64
Nothing to boot: No such file or directory (https://ipxe.org/2d03e13b)
No more network devices

Booting from Floppy...                   
Boot failed: could not read the boot disk

No bootable device.

So with linuxkit seemingly starting a VM successfully, I went to go gather some diagnostics for you guys. Running /opt/docker-desktop/bin/com.docker.diagnose gather made this, and pretty much no other details, spin forever:

[2023-10-10T05:53:21.145890274Z][com.docker.diagnose.ipc][I] (ea9cf16f-285) e573d627-diagnose C->S  GET /ping
[2023-10-10T05:53:21.146462116Z][com.docker.diagnose.ipc][W] (ea9cf16f-285) e573d627-diagnose C<-S NoResponse GET /ping (554.485µs): Get "http://ipc/ping": dial unix <HOME>/.docker/desktop/backend.native.sock: connect: no such file or directory

Checking the available files in that directory got me this:

kim@kimtalkstech:~/.docker/desktop $ ls -la
total 2584
drwxrwxr-x 6 kim kim   4096 Oct 10 01:56  .
drwxrwxr-x 6 kim kim   4096 Oct 10 01:07  ..
-rw-rw-r-- 1 kim kim 790528 Oct 10 01:48  10-10-2023-01-48.zip
-rw-rw-r-- 1 kim kim 847872 Oct 10 01:49  10-10-2023-01-49.zip
-rw-rw-r-- 1 kim kim 964393 Oct 10 02:11  10-10-2023-01-56.zip
-rwxrwxr-x 1 kim kim      0 Oct 10 00:41  backend.lock
srwxrwxr-x 1 kim kim      0 Oct 10 01:07  backend.sock
srwxrwxr-x 1 kim kim      0 Oct 10 01:07  console.sock
srwxrwxr-x 1 kim kim      0 Oct 10 01:07  docker-dev-env-api.sock
srwxrwxr-x 1 kim kim      0 Oct 10 01:07  ethernet-vpnkit.sock
drwxrwxr-x 2 kim kim   4096 Oct 10 00:41  extensions
-rwxrwxr-x 1 kim kim      3 Oct 10 01:07  locked-directories
drwxrwxr-x 4 kim kim   4096 Oct 10 00:41  log
-rwxrwxr-x 1 kim kim   2785 Oct 10 01:07  settings.json
drwxrwxr-x 2 kim kim   4096 Oct 10 01:07  tasks
-rwxrwxr-x 1 kim kim      3 Oct 10 01:07 'unleash-repo-schema-v1-Docker Desktop.json'
-rwxrwxr-x 1 kim kim     37 Oct 10 00:40  userId
drwxrwxr-x 3 kim kim   4096 Oct 10 00:41  vms
srwxrwxr-x 1 kim kim      0 Oct 10 01:07  vpnkit-bridge-fd.sock

So I symlinked (ln -s backend.sock backend.native.sock) in order to get the diag gather tool to complete.

Expected behavior

Docker Engine should just work after a fresh install, and the diagnostic tool shouldn't bug out.

docker version

kim@kimtalkstech:~/.docker/desktop $ docker version
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Client: Docker Engine - Community
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:40 2023
 OS/Arch:           linux/amd64
 Context:           default
kim@kimtalkstech:~/.docker/desktop $ docker compose version
Docker Compose version v2.22.0-desktop.2

### docker info

```bash
kim@kimtalkstech:~/.docker/desktop $ docker info
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.22.0-desktop.2
    Path:     /usr/lib/docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /usr/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /usr/lib/docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.8
    Path:     /usr/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/lib/docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /usr/lib/docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.7
    Path:     /usr/lib/docker/cli-plugins/docker-scout

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

### Diagnostics ID

4ffa772b-1e62-4d50-96f4-fbbe76f29e4c/20231010062447

### Additional Info

kim@kimtalkstech:~/.docker/desktop $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 23.04 Release: 23.04 Codename: lunar kim@kimtalkstech:~/.docker/desktop $ uname -m x86_64

[L-Sypniewski]

@kimdcottrell Have you managed to fix your issue? I have a similar problem: [FAIL] DD0017: can a VM be started? vm has not started: failed to open kmsg.log: open /home/lukasz/.docker/desktop/log/vm/kmsg.log: no such file or directory.

It stopped working when I tried to move Docker.raw location with Docker Desktop GUI. Now I'm not able to make Docker Desktop work despite purging and re-installing it multiple times.

Diagnostics ID: 4ec33231-ee9a-4ef6-b10c-1ab0a5ecaf23/20231220120503

[nate998877]

I have a similar, but probably unrelated issue with [FAIL] DD0017: can a VM be started? vm has not started: vm has not started. It would be nice to see some progress on this to rule out it being related before opening up a ticket of my own.

Diagnostics ID: FB892A41-1B5E-4BD8-A571-94980945D9C5/20240417195755