K8s pod stuck in init with `Error: stat /Users/blah/blub: no such file or directory`

[bschuhmann]

Description

Pod with persistent volume stuck in init with error message: Error: stat /Users/blah/blub: no such file or directory

Persistent volume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: a-pv
spec:
  capacity:
    storage: 1Gi
  hostPath:
    path: /Users/blah/blub
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: a-hostpath
  volumeMode: Filesystem

Storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: a-hostpath
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

All configured via Helm charts. The same cluster configuration has been working until 4.38.0. After upgrading DD to 4.39.0 and restarting the K8s cluster at the end of the update all pods using the persistent volume are stuck.

Workaround: stop DD, downgrade to 4.38.0 and start DD again. K8s starts the cluster and pods start as expected without errors.

Reproduce

Hopefully the information provided in description is enough. Can try to build a stripped down reproducer if necessary.

Expected behavior

See description - the same K8s configuration is running as expected with 4.38.0

docker version

Client:
 Version:           28.0.1
 API version:       1.48
 Go version:        go1.23.6
 Git commit:        068a01e
 Built:             Wed Feb 26 10:38:16 2025
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.39.0 (184744)
 Engine:
  Version:          28.0.1
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.6
  Git commit:       bbd0a17
  Built:            Wed Feb 26 10:40:57 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.2.4
  GitCommit:        v1.2.4-0-g6c52b3f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    28.0.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v0.9.4
    Path:     /Users/blah/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.21.1-desktop.2
    Path:     /Users/blah/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.33.1-desktop.1
    Path:     /Users/blah/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.38
    Path:     /Users/blah/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.5
    Path:     /Users/blah/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/blah/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/blah/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/blah/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/blah/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/blah/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.16.3
    Path:     /Users/blah/.docker/cli-plugins/docker-scout

Server:
 Containers: 82
  Running: 40
  Paused: 0
  Stopped: 42
 Images: 34
 Server Version: 28.0.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 7.654GiB
 Name: docker-desktop
 ID: 3f9a6a1f-d239-4929-b9c7-2b7afd51e672
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/blah/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

209EEDFB-481D-4ACD-873D-06F565DBD84C/20250310124954

Additional Info

No response

[brandondoran]

I'm seeing a similar problem when upgrading from 4.38.0 to 4.39.0

My error is: MountVolume.SetUp failed for volume "bar" : hostPath type check failed: /Users/foo/bar is not a directory

Confirmed that after downgrading to 4.38.0 the problem does not exist

[amanfredi]

At least 4 people on our team report the same issue when updating to 4.39.0

[ericmail84]

I am similarly having issues with containers not having access to directories. I noticed that docker desktop did not have full disk access, but even turning that on didn't fix the issue. Reverting to 4.38.0 fixed the issue.

[misiektg86]

adter upgrading to 4.39.0 on my MBP M4 pro on docker compose up get this: Error response from daemon: failed to populate volume: error while mounting volume '/var/lib/docker/volumes/docker_rabbitmq-data/_data': failed to mount local volume: mount /Users/mk/dev/rabbitmq_data:/var/lib/docker/volumes/docker_rabbitmq-data/_data, flags: 0x1000: no such file or directory directories exists Reverting to 4.38.0 solves the issue

[ties-s]

I also have this issue after upgrading. Also tried giving Docker full disk access permission without success. Regular docker containers can successfully access the folder in question.

MountVolume.SetUp failed for volume "XXXXX" : hostPath type check failed: /Users/xxx/xxx/xxx is not a directory

[cgaspard]

Same issue here. Force to revert back to 4.38.

[adamjenson]

+1

[ahachmann]

+1

[Kyle-Esper]

Is fixed in 4.40.0? My team is stuck on 3.37.2 due to a series of issues in the last few releases

[ericmail84]

I'm definitely having the same issue after 4.40.0.

[bschuhmann]

I'm definitely having the same issue after 4.40.0.

Same here - reverted back to 4.38.

[johnhoran]

Ran into something similar with kubernetes volumes. Changing the volume type from directory to directoryorcreate fixed it, and the directory that got mounted was the pre-existing directory.

[adamjenson]

Ran into something similar with kubernetes volumes. Changing the volume type from directory to directoryorcreate fixed it, and the directory that got mounted was the pre-existing directory.

That does seem to work for regular mounts, but using the command line for options like --set-file fail on versions after 4.38.0

[uriacuesta]

This may have to do with volumes using subpaths. Similar to this issue. https://github.com/kubernetes/minikube/issues/2256

I converted all volumes with subpaths to one not using subpaths and it works fine in v4.40.0.

Can confirm going back to v4.38.0 makes subpath work.

[bschuhmann]

See https://github.com/docker/for-mac/issues/7625#issuecomment-2866011513 - the development build @bsousaa provided, fixes my issue too.

[jwatte]

This fails on 4.41.2 too; I'm seeing it with kubernetes host path mounts.

[SpoddyCoder]

The newly released 4.42.0 (195023) build fixes this issue 👍

[bkowens]

Just tested with 4.42.0 - Still broken. Still says "Directory not found" I have tried both with "/host_mnt" and without on /tmp /Users and /Volumes Every pod reports: hostPath type check failed: DIRECTORY is not a directory

For comparison I tested with docker-compose, works as expected.