for-win
for-win copied to clipboard
Published
20 hours ago •
docker
docker
docker builder failed to call diskusage: DeadlineExceeded: context deadline exceeded
Description
Recently my docker builds started occurring what appears to be disk space issues. The first time I encountered this issue I went into the Docker Desktop UI -> Builds -> Builder Settings and for the currently active builder called desktop-linux I saw a image-like message saying "Storage error" and "docker builder failed to call diskusage: DeadlineExceeded: context deadline exceeded" and "20GB storage limit". The build of one of the services kept failing with "4.746 : You don't have enough free space in /var/cache/apt/archives/.". Since there was an update I went ahead and installed it and now I get some sort of pie chart diagram in the desktop-linux builder info, which again says "Storage limit 20GB", but also "Used 121.9GB", "Shared 117.3GB" with the pie chart reading "59.3MB Local sources" and "121.8GB Regular". Since I'm using the WSL2 integration for Docker Desktop I went into the Debian WSL console and ran df -h, which reports the following:
Filesystem Size Used Avail Use% Mounted on
none 14G 4.0K 14G 1% /mnt/wsl
none 931G 595G 336G 64% /usr/lib/wsl/drivers
none 14G 0 14G 0% /usr/lib/wsl/lib
/dev/sdc 2.9T 1.7T 1.2T 60% /
none 14G 76K 14G 1% /mnt/wslg
rootfs 14G 1.9M 14G 1% /init
none 14G 0 14G 0% /dev
none 14G 4.0K 14G 1% /run
none 14G 0 14G 0% /run/lock
none 14G 0 14G 0% /run/shm
none 14G 0 14G 0% /run/user
tmpfs 14G 0 14G 0% /sys/fs/cgroup
none 14G 76K 14G 1% /mnt/wslg/versions.txt
none 14G 76K 14G 1% /mnt/wslg/doc
drvfs 931G 595G 336G 64% /mnt/c
drvfs 7.3T 1.2T 6.1T 17% /mnt/d
drvfs 931G 235G 696G 26% /mnt/e
drvfs 3.7T 2.4T 1.4T 64% /mnt/f
/dev/sde 251G 238G 177M 100% /mnt/wsl/docker-desktop-data/isocache
drvfs 931G 595G 336G 64% /Docker/host
none 14G 8.0K 14G 1% /mnt/wsl/docker-desktop/shared-sockets/host-services
/dev/sdf 1007G 57M 956G 1% /mnt/wsl/docker-desktop/docker-desktop-user-distro
/dev/loop0 463M 463M 0 100% /mnt/wsl/docker-desktop/cli-tools
I know I have plenty of space on the WSL2 vhdx as I have a 4TB dedicated drive reserved specifically for the WSL virtual disk with 1.32TB of free space left and I see no reason to limit any disk space for purposes of Docker cache, images and volume storage.
I also tried the self-diagnose tool as explained here:
https://docs.docker.com/desktop/troubleshoot/overview/#self-diagnose-tool
In powershell after running
& "C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe" check
the output is
Starting diagnostics
com.docker.diagnose.exe : [2024-09-11T09:40:33.688851700Z][com.docker.diagnose.exe.ipc] c1a5278a-diagnose -> \\.\pipe\dockerBackendApiServer
BackendAPI
At line:1 char:1
+ & "C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe" ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: ([2024-09-11T09:...rver BackendAPI:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
[2024-09-11T09:40:33.699823400Z][com.docker.diagnose.exe.ipc] (21bd8fee) c1a5278a-diagnose C->S BackendAPI POST /idle/make-busy
[2024-09-11T09:40:33.701460000Z][com.docker.diagnose.exe.ipc] (21bd8fee) c1a5278a-diagnose C<-S dc927013-BackendAPI POST /idle/make-busy
(1.6366ms): 0xc0006246f0
[2024-09-11T09:40:34.708119700Z][com.docker.diagnose.exe.ipc] (21bd8fee-1) c1a5278a-diagnose C->S BackendAPI GET /idle
[2024-09-11T09:40:34.711749900Z][com.docker.diagnose.exe.ipc] (21bd8fee-1) c1a5278a-diagnose C<-S dc927013-BackendAPI GET /idle (3.4234ms):
{"apisInFlight":{"/v1.46/images/get from Docker-Client/unknown-version (windows)":2},"booted":true,"busyReason":["2 calls to /v1.46/images/get
from Docker-Client/unknown-version (windows)","timed activities:
map[/idle/make-busy:28991669000]"],"timedActivities":{"/idle/make-busy":"float64"},"vmRunning":true}
[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[SKIP] DD0002: does the bootloader have virtualization enabled?
[SKIP] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[SKIP] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[SKIP] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0024: is WSL installed?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0005: is the user in the docker-users group?
[PASS] DD0038: is the connection to Docker working?
[PASS] DD0014: are the backend processes running?
[PASS] DD0007: is the backend responding?
[SKIP] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[SKIP] DD0006: is the Docker Desktop Service responding?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0002: does the bootloader have virtualization enabled?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[PASS] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0024: is WSL installed?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0032: do Docker networks overlap with host IPs?
No fatal errors detected.
Reproduce
My docker compose stack is quite big (100+ services) and over time there are updates to many of the images and apt packages within the images and it needs to be rebuilt.
-
docker compose build
Expected behavior
docker compose build should succeed
docker version
Client:
Version: 20.10.5+dfsg1
API version: 1.41
Go version: go1.15.15
Git commit: 55c4c88
Built: Mon May 30 18:34:49 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Desktop ()
Engine:
Version: 27.2.0
API version: 1.47 (minimum version 1.24)
Go version: go1.21.13
Git commit: 3ab5c7d
Built: Tue Aug 27 14:15:15 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.20
GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.16.2-desktop.1)
compose: Docker Compose (Docker Inc., v2.29.2-desktop.2)
debug: Get a shell into any image or container (Docker Inc., 0.0.34)
desktop: Docker Desktop commands (Alpha) (Docker Inc., v0.0.15)
dev: Docker Dev Environments (Docker Inc., v0.1.2)
extension: Manages Docker extensions (Docker Inc., v0.2.25)
feedback: Provide feedback, right in your terminal! (Docker Inc., v1.0.5)
init: Creates Docker-related starter files for your project (Docker Inc., v1.3.0)
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
scout: Docker Scout (Docker Inc., v1.13.0)
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory
Server:
Containers: 52
Running: 0
Paused: 0
Stopped: 52
Images: 1592
Server Version: 27.2.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: unconfined
Kernel Version: 5.15.90.1-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 27.41GiB
Name: docker-desktop
ID: 36c7edfc-a3e7-4b9f-9304-729c5228b84a
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Registry: https://index.docker.io/v1/
Labels:
com.docker.desktop.address=unix:///var/run/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile
Diagnostics ID
432BCC77-B0B5-4D17-BCC8-CEA9C3669229/20240911093329
Additional Info
No response
