Space used even if docker system df shows 0 bytes

[frafra]

• [x] This is a bug report
• [ ] This is a feature request
• [X] I searched existing issues before opening this one

Expected behavior

docker system prune -a -f --volumes should remove all the space used by docker volumes and temporary folders and docker system df should show numbers that matches the real disk usage.

Actual behavior

(set -x; docker system prune -a -f --volumes; docker system df; du -hs ~/.local/share/docker/)
+ docker system prune -a -f --volumes
Total reclaimed space: 0B
+ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          0         0         0B        0B
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B
+ du -hs /home/francesco.frassinell/.local/share/docker/
du: cannot read directory '/home/francesco.frassinell/.local/share/docker/overlay2/7df11ffd1929b8280ec82e4d9b7b08d43e43737096e779d41a5d403ec409b7b3/diff/etc/ssl/private': Permission denied
du: cannot read directory '/home/francesco.frassinell/.local/share/docker/overlay2/b12fe35a6e04faf759fedf4b265850add07c31b6d3bab4ca86fa2fa467e4f5f4/diff/run/postgresql/13-main.pg_stat_tmp': Permission denied
du: cannot read directory '/home/francesco.frassinell/.local/share/docker/overlay2/b12fe35a6e04faf759fedf4b265850add07c31b6d3bab4ca86fa2fa467e4f5f4/diff/var/lib/postgresql/13/main': Permission denied
588M	/home/francesco.frassinell/.local/share/docker/

Steps to reproduce the behavior

Difficult to say, I didn't notice that for a while.

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:08 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:16 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 4.15.0-147-generic
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.66GiB
 Name: nixstd04162q003
 ID: CGIW:XGOT:RAWE:LROU:PP5A:FMQF:YA3Z:SSV5:FBTL:XFC2:N277:WATS
 Docker Root Dir: /home/francesco.frassinell/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.)

Docker running in rootless mode.

[mafortin]

I have the same exact issue and I can't find any solution online. Would appreciate an update on that.

[smheidrich]

Yeah this is really annoying. It just leaves these files hanging around. There is a StackOverflow answer from 2020 where someone had the same issue with the VFS dirs under /var/lib/docker and just removed them manually after verifying that there weren't any images/volumes that could be using them. Curiously, I don't have any VFS dirs under /var/lib/docker, so maybe the issue has been fixed for /var/lib/docker since then but not ~/.local/share/docker?