Constantly growing /var/lib/docker/overlay2/

[arnegroskurth]

Description

We have a set of job-runners that perform various build- and deployment-related tasks using docker. After some time (weeks), all of them end up with critical disk usage and have to be re-provisioned as there does not appear to be a way to release the used disk space via the docker CLI:

$ du -sh /var/lib/docker/*
72.4M	/var/lib/docker/buildkit
71.2M	/var/lib/docker/containers
4.0K	/var/lib/docker/engine-id
293.9M	/var/lib/docker/image
132.0K	/var/lib/docker/network
65.7G	/var/lib/docker/overlay2
0	/var/lib/docker/plugins
0	/var/lib/docker/runtimes
0	/var/lib/docker/swarm
0	/var/lib/docker/tmp
5.0G	/var/lib/docker/volumes
$ ls -l /var/lib/docker/overlay2/ | wc -l
4204
$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          2         2         123.7MB   0B (0%)
Containers      2         2         168B      0B (0%)
Local Volumes   62        0         5.292GB   5.292GB (100%)
Build Cache     0         0         0B        0B
$ docker builder ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default * docker                                       
  default default         running v0.11.7+d3e6c1360f6e linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
$ docker builder du
Reclaimable:	0B
Total:		0B
$ docker system prune -a -f
Total reclaimed space: 0B
$ docker builder prune -a -f
Total:	0B

Related to #32420, #43586

Reproduce

I'm not sure as it appears to be building up slowly over time.

Expected behavior

The used disk space under /var/lib/docker/overlay2/ shows up in any diagnostic docker ... command and can be released with any (other) docker ... command. I would expect it to show up under docker system df and/or docker builder du and for it t be releaseable with docker system prune or docker builder prune.

docker version

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:09:13 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-70.13.1.el9_0.x86_64
 Operating System: Rocky Linux 9.1 (Blue Onyx)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.28GiB
 Name: xxx
 ID: c603ff4f-a1fb-4a1c-b323-75ef6e588528
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: xxx
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response