Desktop docker doesn't work (again?)

[farcaller]

Bug description

$ docker ps
fork/exec /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli: no such file or directory

To Reproduce

Steps to reproduce the behavior:

this is docker 20.10.21, windows 11, nixos-wsl from 522219248de4b5876f18e47f34d979dd9f4fcbdc.

Expected behavior Docker runs.

Logs

$ ls -l /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli
Unable to determine time zone: No such file or directory (os error 2)
.rwxr-xr-x 51M root 25 Oct 18:00 /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli
$ ldd /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli
        linux-vdso.so.1 (0x00007ffc71f8f000)
        libpthread.so.0 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libpthread.so.0 (0x00007fd217684000)
        libdl.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libdl.so.2 (0x00007fd21767f000)
        libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007fd217476000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007fd21a0c0000)
$  /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli

Usage:  docker [OPTIONS] COMMAND

A self-sufficient runtime for containers
...

[573]

Cannot reproduce on 522219248de4b5876f18e47f34d979dd9f4fcbdc

$ nix flake metadata | grep nixos-wsl
├───nixos-wsl: github:nix-community/NixOS-WSL/522219248de4b5876f18e47f34d979dd9f4fcbdc

$ rg -B 8 -A 3 docker-native
nixos/base/nixoswsl.nix
17-    wsl = {
18-      enable = true;
19-      defaultUser = "573";
20-      interop = {
21-        register = true;
22-     preserveArgvZero = true;
23-      };
24-      nativeSystemd = true;
25:      docker-native = {
26-        enable = true;
27-      };
28-    };

$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES $ readlink $(which docker) /nix/store/h9h695l84fpnv10qd0q2vqxzzknpjhqx-docker-20.10.21/bin/docker

[nzbr]

The title says that this is about docker-native, but your error log shows a message from docker-desktop. Can you send your docker-related NixOS settings?

[farcaller]

Sorry, I mixed up the native and desktop dockers. Meant to be the latter.

  wsl = {
    enable = true;
    wslConf.automount.root = "/mnt";
    defaultUser = "nixos";
    startMenuLaunchers = true;
    wslConf.network.hostname = "wsl";
    docker-desktop.enable = true;
    nativeSystemd = true;
  };

[daniel-chambers]

I also have encountered the problem. I think the issue is because the com.docker.cli executable in the docker desktop mount is dynamically linked. If you run

nix-shell -p file --run "file /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli"

You will get:

/mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=13446ee25c78f218065b1ccffc127182fe2b7008, for GNU/Linux 3.2.0, not stripped

/lib64/ld-linux-x86-64.so.2 is not available on NixOS. I think that's why we get the rather misleading "no such file or directory" error when trying to execute com.docker.cli.

Versions:

• Docker Desktop 4.16.3
• WSL 1.0.3
• Windows 10 22H2

I'm running the NixOS-WSL distro built from main (https://github.com/nix-community/NixOS-WSL/commit/65a6b5aeda9e454d3cd9d75648f59333a0094122).

My configuration.nix is the defaults plus docker-desktop.enable = true and nativeSystemd = true:

{ pkgs, config, modulesPath, ... }:

let
  nixos-wsl = import ./nixos-wsl;
in
{
  imports = [
    nixos-wsl.nixosModules.wsl
  ];

  wsl = {
    enable = true;
    wslConf.automount.root = "/mnt";
    defaultUser = "nixos";
    startMenuLaunchers = true;

    # Enable native Docker support
    # docker-native.enable = true;

    # Enable integration with Docker Desktop (needs to be installed)
    docker-desktop.enable = true;

    nativeSystemd = true;
  };

  # Enable nix flakes
  nix.package = pkgs.nixFlakes;
  nix.extraOptions = ''
    experimental-features = nix-command flakes
  '';

  system.stateVersion = "22.11";
}

[nzbr]

Is com.docker.cli just the docker command or the api proxy as well? Until it broke at least, the api proxy was statically linked and we just used the docker cli from nix instead of the one provided by docker-desktop

[daniel-chambers]

I think it's just the docker command. It appears that running /mnt/wsl/docker-desktop/cli-tools/usr/bin/docker causes that executable to try to run com.docker.cli in the same directory, which is where things crash with the OP error. Not sure how that all works tbh, they're all opaque binary executables.

However, I did a little more digging and it seems that the docker proxy service that the wsl.docker-desktop.enable integration installs into systemd creates a tonne of symlinks all over the disk. Using journald -u docker-desktop-proxy.service, I can see logs like:

creating symlink for /usr/bin/docker
creating symlink for /usr/bin/docker-compose
creating symlink for /usr/bin/docker-compose-v1
creating symlink for /usr/bin/docker-credential-desktop.exe
creating symlink for /usr/bin/docker-index
creating symlink for /usr/bin/hub-tool
creating symlink for /usr/local/bin/kubectl

So when I do which docker, I find I'm actually running /usr/bin/docker, which is a symlink to /mnt/wsl/docker-desktop/cli-tools/usr/bin/docker.

However, when I run which -a docker, I see there's actually multiple dockers on the path:

/usr/bin/docker
/mnt/c/Program Files/Docker/Docker/resources/bin/docker
/run/current-system/sw/bin/docker

And if I run /run/current-system/sw/bin/docker, that works and connects successfully to Docker Desktop (I assume via the proxy)!

So it appears the core problem is that the docker proxy service is messing up the path by putting its own dynamically linked docker exes on there, overriding the statically linked ones that have already been installed. Is this new, unexpected, behaviour?

[nzbr]

I believe it did not create the /usr/bin symlinks before

[K900]

Is this still an issue?

[nzbr]

Nobody fixed it, so I think yes. Docker Desktop is annoying to test with aunfortunately. (I'm happy if I get it to run at all)

[zevisert]

I was able to get docker desktop connected again, basically it seems like docker desktop is expecting a FHS in the distros it's connecting to. Parsing the log from the docker proxy shows what it was trying to do:

# "C:\Users\zevisert\AppData\Local\Docker\log\host\com.docker.wsl-distro-proxy.exe.log"
... 
[2023-12-19T21:55:39.966680700Z][com.docker.wsl-distro-proxy.exe][I] start wsl integration for distro nixos
[2023-12-19T21:55:39.969885900Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e sh -c mkdir -p ~/.docker/run
[2023-12-19T21:55:40.075811600Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e sh -c cat ~/.docker/config.json
[2023-12-19T21:55:40.183053500Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e sh -c cat - > ~/.docker/config.json
[2023-12-19T21:55:40.291340800Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e sh -c mkdir -p ~/.docker/scan
[2023-12-19T21:55:40.416069200Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e sh -c cat ~/.docker/scan/config.json
[2023-12-19T21:55:40.556760300Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e sh -c cat - > ~/.docker/scan/config.json
[2023-12-19T21:55:40.680547000Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e whoami
[2023-12-19T21:55:40.788894800Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -u root -e sh -c ( addgroup docker || true ) && addgroup "zevisert" docker
[2023-12-19T21:55:40.896861100Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e cat /etc/wsl.conf
[2023-12-19T21:55:41.053001800Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e cat /etc/wsl.conf
[2023-12-19T21:55:41.161760400Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e /mnt/wsl/docker-desktop/docker-desktop-user-distro sync-context-store /mnt/c/Users/zevisert/.docker/contexts
[2023-12-19T21:55:41.269060500Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e cat /etc/wsl.conf
[2023-12-19T21:55:41.378134200Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e cat /etc/wsl.conf
[2023-12-19T21:55:41.500978800Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e /mnt/wsl/docker-desktop/docker-desktop-user-distro sync-home-subdir /mnt/c/Users/zevisert/.aws
[2023-12-19T21:55:41.610026000Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e cat /etc/wsl.conf
[2023-12-19T21:55:41.719162900Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e cat /etc/wsl.conf
[2023-12-19T21:55:41.827339600Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e /mnt/wsl/docker-desktop/docker-desktop-user-distro sync-home-subdir /mnt/c/Users/zevisert/.azure
[2023-12-19T21:55:41.937073700Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e cat /etc/wsl.conf
[2023-12-19T21:55:42.045041700Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e cat /etc/wsl.conf
[2023-12-19T21:55:42.153630500Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e /mnt/wsl/docker-desktop/docker-desktop-user-distro sync-file /mnt/c/Users/zevisert/.docker/features.json ~/.docker/features.json
[2023-12-19T21:55:42.327061000Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] Killing CLI API server running at /home/zevisert/.docker/run/docker-cli-api.sock
[2023-12-19T21:55:42.329657500Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e ls /home/zevisert/.docker/run/docker-cli-api.sock
[2023-12-19T21:55:42.434772200Z][com.docker.wsl-distro-proxy.exe][I] Starting WSL Distro proxy 2
[2023-12-19T21:55:42.434772200Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e cat /etc/wsl.conf
[2023-12-19T21:55:42.545078300Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -u root -e /mnt/wsl/docker-desktop/docker-desktop-user-distro proxy --distro-name nixos --docker-desktop-root /mnt/wsl/docker-desktop C:\Program Files\Docker\Docker\resources --show-kube-system-containers=true
[2023-12-19T21:55:42.627120500Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] Starting distro cli api server on /home/zevisert/.docker/run/docker-cli-api.sock
[2023-12-19T21:55:42.627120500Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d nixos -e docker serve --address unix:///home/zevisert/.docker/run/docker-cli-api.sock

To get this all to work, I used the wsl.extraBin option in my /etc/nixos/configuration.nix:

{ config, lib, pkgs, ... }: {
  wsl = {
    enable = true;
    defaultUser = "zevisert";
    extraBin = with pkgs; [
      # Binaries for Docker Desktop wsl-distro-proxy
      { src = "${coreutils}/bin/mkdir"; }
      { src = "${coreutils}/bin/cat"; }
      { src = "${coreutils}/bin/whoami"; }
      { src = "${coreutils}/bin/ls"; }
      { src = "${busybox}/bin/addgroup"; }
      { src = "${su}/bin/groupadd"; }
      { src = "${su}/bin/usermod"; }
    ];
  }; 
  # Other options ...
}