Volumes not initially mounting for services that restart on windows reboot

[SpikyClip]

Description

I am having the exact same issue as #584 on Docker Desktop 4.28.0 (139021) for Windows.

Essentially on computer boot, when the service restarts, the containers cannot find the volumes so it appears like its the first setup. But when the services are restarted manually, the volumes are mounted with no issues.

This make startup services useless as they can't access the volumes they need. I suspect it has something to do with delayed WSL2 startup, or the mount paths I used in the compose file.

docker-compose_truncated.yml.txt

Reproduce

sudo docker-compose up -d

Expected behavior

Volumes should be mounted correctly when docker desktop starts on windows reboot.

docker version

Client:
 Cloud integration: v1.0.35+desktop.11
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.22
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.5.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 7
  Running: 6
  Paused: 0
  Stopped: 1
 Images: 7
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 7.734GiB
 Name: docker-desktop
 ID: a81fab99-f491-41fa-8fc1-09bc414d91f9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

E82A5D0F-B3D8-4BED-9935-9D986C6BB0D8/20240322090128

Additional Info

No response

[Tristan-MyAnaPro]

@SpikyClip / @bsousaa Possibly a duplicate of the issue I encounter too ? https://github.com/docker/for-win/issues/13947

[evilworm]

I'm experiencing the same issue. I have mariadb instance which mounts a local folder to /var/lib/mysql ... when I boot the computer, it does not mount the folder correctly although docker inspect shows it as mounted. But when you docker-compose exec into the container, I can see the default directory structure for freshly launched mariadb instance. After restarting the container, the folder is mounted correctly.

[yolopunk]

I'm having the same problem with docker-desktop 4.28.0, the volume mount fails on a windows reboot, but it works fine on a manual startup. When I roll back the version of docker-desktop to 4.20.0, I don't have this problem.

[SpikyClip]

I tried downgrading to 4.24.2 and and it works. So the bug is between that version and 4.28. Don't really feel like testing all the version inbetween, but feel free to chime in if any later version works.

[barbuslex]

I have same issue in 4.30.0 :S

4.24.2 is dated of 12-10-2023... need fix please...

[nikhil96widhani]

Same issue for me, i downgraded for now but hope its fixed in next version

[longzheng]

I'm running into this bug on Windows as well with the Powerwall-Dashboard container (although it doesn't seem to affect my other Teslamate container).

I tried downgrading to 4.24.2 and and it works. So the bug is between that version and 4.28. Don't really feel like testing all the version inbetween, but feel free to chime in if any later version works.

I can also verify that 4.24.2 is working correctly.

I tested different versions all the way up to 4.27.2 and it's still working so I think it's literally 4.28.0 that broke it.

[longzheng]

I can confirm that version 4.31.0 still has the same bug/issue.

[longzheng]

Looking at https://github.com/dockur/dnsmasq/issues/4, I also checked my dockerd daemon logs at %LOCALAPPDATA%\Docker\log\vm\dockerd.log and confirmed it's the same issue with bind mounts

time="2024-06-07T06:45:38.687149000Z" level=error msg="failed to start container" container=c47bbcc2866be37edb367e957c29f2c761c813d853e0119d76baf07a9c4a0377 error="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/f8bdc6a9944521e050c4ef3a1e2ae3b2b39eb573242081361afb80a0c5a649e7\" to rootfs at \"/etc/telegraf/telegraf.conf\": stat /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/f8bdc6a9944521e050c4ef3a1e2ae3b2b39eb573242081361afb80a0c5a649e7: no such file or directory: unknown"
time="2024-06-07T06:45:38.871371700Z" level=error msg="failed to start container" container=843603cdcb15df27d3a8d374da931aa905fdfe695a80ee0686f2f86538460b79 error="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/c964d0391d417d8719c015cbb73a53a85e64860f8fd33daa854b4eb49de7fd52\" to rootfs at \"/var/lib/grafana\": stat /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/c964d0391d417d8719c015cbb73a53a85e64860f8fd33daa854b4eb49de7fd52: no such file or directory: unknown"
time="2024-06-07T06:45:39.000972200Z" level=error msg="failed to start container" container=f89a22df31829d4870f8a0396148a0b1227e1b8431f05e8476128c58ee84ffb0 error="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/a6326bcc0b9c72f750ab2a5f664b17f7c9a14530afa144b290774d0123ab3d6f\" to rootfs at \"/etc/influxdb/influxdb.conf\": stat /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/a6326bcc0b9c72f750ab2a5f664b17f7c9a14530afa144b290774d0123ab3d6f: no such file or directory: unknown"
time="2024-06-07T06:45:39.151204500Z" level=error msg="failed to start container" container=7d70b931b3924b604c1855243bcae5d2ffbd14647b84a223787a29b8edd7eb4a error="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/2a2e53533e8c38a8fd40914c28b7fb037cb8f187499a3ff184eb9cc384acb143\" to rootfs at \"/app/.auth\": stat /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/2a2e53533e8c38a8fd40914c28b7fb037cb8f187499a3ff184eb9cc384acb143: no such file or directory: unknown"
time="2024-06-07T06:45:39.291634600Z" level=error msg="failed to start container" container=87ba2bebdd9d755e89d7741fc4ff271ef6d088418574c5cef267b3e423fa5cad error="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/2af5984fa649aa521d7080a30d0e3da96686aa85c0ec42566802c195d1177cf4\" to rootfs at \"/var/lib/weather\": stat /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/2af5984fa649aa521d7080a30d0e3da96686aa85c0ec42566802c195d1177cf4: no such file or directory: unknown"
time="2024-06-07T06:45:39.291928300Z" level=info msg="Loading containers: done."

I concur that this smells like a race condition with WSL2 when using bind mounts.

[kanjieater]

Just want to call out that a side effect to the bind mounts being created too early, seems to resulting in docker not seeing that the folder structures already exists, which if you're using the docker compose shorthand syntax

volumes:
	- /path/to/config:/config

instead of

    volumes:
      - type: bind
        source: /path/to/config
        target: /config

Docker will always create the path prior to the container as root, which causes all kinds of havoc to your permissions if you're trying to run as the usual UID:GID, 1000:1000.

[longzheng]

After hours of trial and error testing, I have narrowed down this problem specifically to just bind mounting files (not folders) and using WSL2 for docker-compose.

Reproduction

The simplest repro I came up with was basically a docker-compose.yml with just one service and an individual file mount.

services:
  influxdb:
    image: influxdb:1.8
    restart: always
    ports:
      - "8086:8086"
    volumes:
      - "./influxdb.conf:/influxdb.conf"
      # alternatively can use the type: bind syntax which has the same issue
      # - type: bind
      #   source: ./influxdb.conf
      #   target: /influxdb.conf

Then just create a blank called influxdb.conf.

The most important step to reproduce this issue is to initialize the docker container using WSL. So first type wsl to open WSL2 at the current directory, and then docker compose up -d.

This will create the container with a bind mount of /mnt/c/Users/longz/Desktop/test/influxdb.conf.

Then if you log out or restart and log back in, the container will have a status of Excited (127) when Docker Desktop is configured to "start when you sign in to your computer".

Workaround

I found as a workaround, if you start the docker compose from Windows (command prompt), not WSL2, then it will work correctly.

For example from the same directory just run cmd and then docker compose up -d, it will create a container with a bind mount of C:\Users\longz\Desktop\test\influxdb.conf.

This container will start correctly if you log out and log back in.

Logs

When it fails with WSL bind mount

time="2024-06-08T07:46:10.132502900Z" level=info msg="Starting up"
time="2024-06-08T07:46:10.202880400Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
time="2024-06-08T07:46:10.264388100Z" level=info msg="Loading containers: start."
time="2024-06-08T07:46:10.373628100Z" level=info msg="Removing stale sandbox ee9fba78558b36cde4ca5affff4fcca2d893a88ffbfb8429427aefd28ef3497d (f1d46c06e9dd76be4cb6684d52bd57836656ae9ec97198999dde453829ff555c)"
time="2024-06-08T07:46:10.390831000Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 9c75cf86e2840d2e6be23069440acb96a53bbe33a624a326d4fab1d18a8c12a7 e3cd2cbe49631a54b53dd8c2aecf07581fee5cf7cb55f59447dcd649e7d28234], retrying...."
time="2024-06-08T07:46:10.435127800Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2024-06-08T07:46:10.594770200Z" level=error msg="stream copy error: reading from a closed fifo"
time="2024-06-08T07:46:10.594825700Z" level=error msg="stream copy error: reading from a closed fifo"
time="2024-06-08T07:46:10.612752400Z" level=error msg="f1d46c06e9dd76be4cb6684d52bd57836656ae9ec97198999dde453829ff555c cleanup: failed to delete container from containerd: container \"f1d46c06e9dd76be4cb6684d52bd57836656ae9ec97198999dde453829ff555c\" in namespace \"moby\": not found"
time="2024-06-08T07:46:10.806178200Z" level=error msg="failed to start container" container=f1d46c06e9dd76be4cb6684d52bd57836656ae9ec97198999dde453829ff555c error="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/c2760049fc8d5840d923e884d0d5c597983b14a35a07dd23e3fce9b8f080fc27\" to rootfs at \"/influxdb.conf\": mount /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/c2760049fc8d5840d923e884d0d5c597983b14a35a07dd23e3fce9b8f080fc27:/influxdb.conf (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"
time="2024-06-08T07:46:10.806268600Z" level=info msg="Loading containers: done."
time="2024-06-08T07:46:10.817468500Z" level=warning msg="WARNING: No blkio throttle.read_bps_device support"
time="2024-06-08T07:46:10.817501300Z" level=warning msg="WARNING: No blkio throttle.write_bps_device support"
time="2024-06-08T07:46:10.817508100Z" level=warning msg="WARNING: No blkio throttle.read_iops_device support"
time="2024-06-08T07:46:10.817511200Z" level=warning msg="WARNING: No blkio throttle.write_iops_device support"
time="2024-06-08T07:46:10.817513800Z" level=warning msg="WARNING: daemon is not using the default seccomp profile"
time="2024-06-08T07:46:10.817526600Z" level=info msg="Docker daemon" commit=de5c9cf containerd-snapshotter=false storage-driver=overlay2 version=26.1.4
time="2024-06-08T07:46:10.817779000Z" level=info msg="Daemon has completed initialization"
time="2024-06-08T07:46:10.922223200Z" level=info msg="API listen on /var/run/docker.sock"
time="2024-06-08T07:46:10.922306400Z" level=info msg="API listen on /run/guest-services/docker.sock"
time="2024-06-08T07:46:11.884586300Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=9734f53b5c010f77 traceID=70a85b04dae6b8e1eed61279492a76ab
time="2024-06-08T07:51:22.402821015Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=809fa9480d55520e traceID=98f750795508c8218c910caedc28f803
time="2024-06-08T07:51:22.406775154Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=e430ad3d6c1f2dc4 traceID=a5b8a96e770a274da06ceb5a6e91faa6
time="2024-06-08T07:51:22.411005403Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=2e503f3c80cd35fc traceID=491ad4e2ea75f2c13a88bbd50a1afdcf
time="2024-06-08T07:51:22.414956343Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=2801fdf650d385ba traceID=51d82f952b6d1b1a41351a7aeac544e1
time="2024-06-08T07:51:22.418894181Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=07566d24fad9a43e traceID=fff263ebb8faa89bc8359defc738243b
time="2024-06-08T07:51:22.423088929Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=63e06060e803f29a traceID=ce1a9e8048ebaaf80e66637012c8e4de
time="2024-06-08T07:51:22.426915364Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=67f6a0ea5cf3ddd5 traceID=b59eeb439bd14e430b3549f9d709fd4a
time="2024-06-08T07:51:22.431097212Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=5375960a670549ff traceID=8cd07ea66a2ae145aa20c4d16c24e335
time="2024-06-08T07:51:22.435321061Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=0915dc967b22d00d traceID=65527c08ea4ef391692c0a055c860f5b
time="2024-06-08T07:51:22.439222499Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=7cbfc67b3b663796 traceID=b08a0b3ec66a9c1fac2aa9a685455749
time="2024-06-08T07:51:45.128346791Z" level=info msg="ignoring event" container=f1d46c06e9dd76be4cb6684d52bd57836656ae9ec97198999dde453829ff555c module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2024-06-08T07:51:45.133030458Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=f1d46c06e9dd76be4cb6684d52bd57836656ae9ec97198999dde453829ff555c daemonShuttingDown=false error="restart canceled" execDuration=46.080874533s exitStatus="{0 2024-06-08 07:51:45.123416215 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
time="2024-06-08T07:51:45.441325975Z" level=error msg="Error setting up exec command in container f1d46c06e9dd76be4cb6684d52bd57836656ae9ec97198999dde453829ff555c: container f1d46c06e9dd76be4cb6684d52bd57836656ae9ec97198999dde453829ff555c is not running" spanID=42a5f1be7da194e9 traceID=2177681882773035bfc111be4e726530
time="2024-06-08T07:52:13.687265926Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=0e7a632f8f142946 traceID=33e1215940e158c1c3dde3d350cf7e37

When it works with Windows bind mount

time="2024-06-08T07:53:02.925570900Z" level=info msg="Starting up"
time="2024-06-08T07:53:02.988704100Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
time="2024-06-08T07:53:03.055067900Z" level=info msg="Loading containers: start."
time="2024-06-08T07:53:03.157277400Z" level=info msg="Removing stale sandbox 3d85c11f9c934d488f10d1faaa333eac7ef6426463253ad3c79252bfb40f1e78 (e50abc924a28470c95bfda22bbc72d7957eda01e3fa43a3a156bf9a869b819c8)"
time="2024-06-08T07:53:03.174143800Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 9c75cf86e2840d2e6be23069440acb96a53bbe33a624a326d4fab1d18a8c12a7 c24396f04d3dc71a80e5f6b27dc8057000cc6d00445a2d492815297daffb0028], retrying...."
time="2024-06-08T07:53:03.221468600Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2024-06-08T07:53:03.681588800Z" level=info msg="Loading containers: done."
time="2024-06-08T07:53:03.694146000Z" level=warning msg="WARNING: No blkio throttle.read_bps_device support"
time="2024-06-08T07:53:03.694185400Z" level=warning msg="WARNING: No blkio throttle.write_bps_device support"
time="2024-06-08T07:53:03.694194000Z" level=warning msg="WARNING: No blkio throttle.read_iops_device support"
time="2024-06-08T07:53:03.694198900Z" level=warning msg="WARNING: No blkio throttle.write_iops_device support"
time="2024-06-08T07:53:03.694203000Z" level=warning msg="WARNING: daemon is not using the default seccomp profile"
time="2024-06-08T07:53:03.694220200Z" level=info msg="Docker daemon" commit=de5c9cf containerd-snapshotter=false storage-driver=overlay2 version=26.1.4
time="2024-06-08T07:53:03.694458700Z" level=info msg="Daemon has completed initialization"
time="2024-06-08T07:53:03.843328700Z" level=info msg="API listen on /var/run/docker.sock"
time="2024-06-08T07:53:03.843348300Z" level=info msg="API listen on /run/guest-services/docker.sock"
time="2024-06-08T07:53:05.140474500Z" level=error msg="Handler for GET /v1.45/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." spanID=0cd989d415ce81bc traceID=4203602522acced71f2b593de56cf083

[does1tmatter]

I tried downgrading to 4.24.2 and and it works. So the bug is between that version and 4.28. Don't really feel like testing all the version inbetween, but feel free to chime in if any later version works.

Tried 4.26.0 seems to be working fine. So between 4.26 - 4.28 (just wanted to try first 2024 release at least for the time being)