moby
moby copied to clipboard
moby
Breaking change v29.0.0 - containerd snapshotter migration fails on unrecognized graph driver
Description
Apologies for any inconsistent terminology, new to low level docker config.
NB the code links might not work unless the daemon.go file in the PR is already in the expanded state, which it isn't by default, as it's a large file 🫠
When requesting a storage-driver that is not a valid containerd snapshotter, migration is not disabled. Instead, containerd snapshotting is enabled, then immediately fails, as the driver is not a valid containerd snapshotter.
What I think is happening, based on the PR that added this change:
- [link] we initially try to get the graphdriver from
DOCKER_DRIVER/DOCKER_GRAPHDRIVER - [link] we build a layerstore based on the requested driver
- [link] we use this to build the image service config
- Old code for setting
d.imageService- [link] previously, this image service config was (always?) used to build the image service
- New code for setting
d.imageService- [link] we don't set
d.imageServiceunless we are past the migration threshold (we will not be past the threshold due to the earlier check here) - [link] so then we fall through to the
d.imageService == nilpath, starting containerd snapshotter by default - [link] our graphdriver was never mapped to a valid post-migration value, but this validation doesn't check for whether we should migrate or not (see here), so it just fails
- [link] we don't set
I think that the code needs to also check for migration status before enabling containerd snapshotting (this line).
Reproduce
# set storage driver to fuse-overlayfs
[my_user@host ~]$ cat ~/.config/docker/daemon.json
{
"storage-driver": "fuse-overlayfs"
}
# ensure fuse-overlayfs is installed
[my_user@host ~]$ yum info fuse-overlayfs
Last metadata expiration check: 0:00:07 ago on Wed Dec 3 11:42:45 2025.
Installed Packages
Name : fuse-overlayfs
Version : 1.14
Release : 1.el9
Architecture : x86_64
Size : 135 k
Source : fuse-overlayfs-1.14-1.el9.src.rpm
Repository : @System
From repo : appstream
Summary : FUSE overlay+shiftfs implementation for rootless containers
URL : https://github.com/containers/fuse-overlayfs
License : GPLv3+
Description : FUSE overlay+shiftfs implementation for rootless containers.
# ensure docker v29.0.0
[my_user@host ~]$ XDG_RUNTIME_DIR=/run/user/$(id -u) dockerd-rootless.sh --version
(...)
Docker version 29.0.0, build d105562
# no containers/images exist
[my_user@host ~]$ docker images
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
[my_user@host ~]$ docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
# start rootless docker
[my_user@host ~]$ XDG_RUNTIME_DIR=/run/user/$(id -u) dockerd-rootless.sh
(...)
INFO[2025-12-03T11:34:59.831473404+01:00] Starting daemon with containerd snapshotter integration enabled
INFO[2025-12-03T11:34:59.833928108+01:00] stopping healthcheck following graceful shutdown module=libcontainerd
INFO[2025-12-03T11:34:59.833969474+01:00] stopping event stream following graceful shutdown error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: configured driver "fuse-overlayfs" not available: unavailable
[rootlesskit:child ] error: command [/bin/dockerd-rootless.sh] exited: exit status 1
[rootlesskit:parent] error: child exited: exit status 1
Expected behavior
Expected behaviour is that no migration happens if the requested storage driver is not a known compatible containerd snapshotter.
docker version
Client: Docker Engine - Community
Version: 29.0.0
API version: 1.52
Go version: go1.25.4
Git commit: 3d4129b
Built: Mon Nov 10 21:49:09 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 29.0.0
API version: 1.52 (minimum version 1.44)
Go version: go1.25.4
Git commit: d105562
Built: Mon Nov 10 21:45:55 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v2.1.5
GitCommit: fcd43222d6b07379a4be9786bda52438f0dd16a1
runc:
Version: 1.3.3
GitCommit: v1.3.3-0-gd842d771
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client: Docker Engine - Community
Version: 29.0.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.29.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.40.3
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 29.0.0
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: fcd43222d6b07379a4be9786bda52438f0dd16a1
runc version: v1.3.3-0-gd842d771
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.14.0-503.11.1.el9_5.x86_64
Operating System: AlmaLinux 9.5 (Teal Serval)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 62.27GiB
Name: opamuxvm1828.optiver.com
ID: 99cb49c7-fdd0-476d-bc2f-9761425744f2
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http://http-proxy.common.optiver.com:3128
HTTPS Proxy: http://http-proxy.common.optiver.com:3128
No Proxy: optiver.com,optiver.us,git,chigit,trd-chigit,chijenkins
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Firewall Backend: iptables
Additional Info
No response
Could you try on the latest version? We had a similar report already https://github.com/moby/moby/issues/51475 and the fix got merged in 29.1
