cli
cli copied to clipboard
docker
Misleading error about no space left
Description
When building, the error ERROR: failed to solve: failed to read dockerfile: failed to create temp dir: mkdir /var/lib/docker/tmp/buildkit-mount1596176744: no space left on device appears, stating there is not space left on the device. This is misleading, since simply creating /var/lib/docker and granting write access solves the issue.
Reproduce
It started appearing at some point today, after performing builds successfully before, using
$ docker build -t foo:bar .
Expected behavior
The build would just work as it did before.
docker version
Client:
Cloud integration: v1.0.35+desktop.5
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:04:20 2023
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.26.1 (131620)
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:08:15 2023
OS/Arch: linux/arm64
Experimental: true
containerd:
Version: 1.6.25
GitCommit: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Version: 24.0.7
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.0-desktop.2
Path: /Users/xxx/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.23.3-desktop.2
Path: /Users/xxx/.docker/cli-plugins/docker-compose
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /Users/xxx/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.21
Path: /Users/xxx/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: 0.1
Path: /Users/xxx/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v0.1.0-beta.10
Path: /Users/xxx/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/xxx/.docker/cli-plugins/docker-sbom
scan: Docker Scan (Docker Inc.)
Version: v0.26.0
Path: /Users/xxx/.docker/cli-plugins/docker-scan
scout: Docker Scout (Docker Inc.)
Version: v1.2.0
Path: /Users/xxx/.docker/cli-plugins/docker-scout
Server:
Containers: 28
Running: 7
Paused: 0
Stopped: 21
Images: 16
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc version: v1.1.10-0-g18a0cb0
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.5.11-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 8
Total Memory: 9.468GiB
Name: docker-desktop
ID: ff68179a-2616-471c-8ee2-d0fc350a3a21
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 120
Goroutines: 111
System Time: 2024-02-01T18:47:00.138584513Z
EventsListeners: 10
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: true
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
Additional Info
Mac M3 Max on Sonoma 14.2.1 (23C71)
Thanks for reporting!
since simply creating
/var/lib/dockerand granting write access solves the issue.
Can you provide more information on that? Do you mean creating that directory on your macOS host? (so not inside the Docker Desktop VM)? (If that's the case, that'd be ... surprising, as these files should AFAIK all be handled inside the VM 🤔 but perhaps there's some client-side code that tries to use the same location 🤔)
It started appearing the other day out of nowhere, literally. It happened on one of the two following docker files, I don't remember exactly. I am not a Docker pro, but it doesn't look like anything crazy I guess:
# syntax=docker/dockerfile:1
ARG NODE_VERSION=18.17.1
FROM node:${NODE_VERSION}-alpine
RUN apk update && apk add --no-cache build-base gcc autoconf automake zlib-dev libpng-dev nasm bash vips-dev git curl
ENV developOrDevelopMaster develop
ARG NODE_ENV=development
ENV NODE_ENV=${NODE_ENV}
WORKDIR /usr/src/app
COPY . .
RUN npm install --force
EXPOSE 1337
EXPOSE 1338
CMD npm run $developOrDevelopMaster
ARG NODE_VERSION=18.17.1
FROM node:${NODE_VERSION}-alpine
ENV serveLocalChildOrMaster serveLocalChild
WORKDIR /usr/src/app
COPY . .
RUN npm install -g @angular/cli
EXPOSE 4200
EXPOSE 4201
CMD npm run $serveLocalChildOrMaster
After long debugging, I found this one, which is not really my issue I guess, but it hinted me to check for the existence of that folder: https://stackoverflow.com/questions/45812401/no-space-left-on-device-even-after-removing-all-containers I created that dir on my host machine, and things started rolling again. 🤷♂️
I'm getting this issue more often now, to the extent that some build operations just can't be performed, as the necessary cache paths can't be created, and I can't create them myself beforehand. Very frustrating. Why can't docker just write the directories it needs?
A few examples:
Dockerfile:16
--------------------
14 | #ENV NODE_ENV production
15 |
16 | >>> WORKDIR /usr/src/app
17 |
18 | # Copy the rest of the source files into the image.
--------------------
ERROR: failed to solve: failed to compute cache key: mount callback failed on /tmp/containerd-mount1749269262: write /tmp/containerd-mount1749269262/usr/bin/cksum: no space left on device
I have over 700 gigs
Seems you're using Docker Desktop, do you have a diagnostic ID to share? (https://docs.docker.com/desktop/troubleshoot/overview/#diagnose). For better traceability https://github.com/docker/for-mac to report this kind of issue would be better but let's check here first.
I also just received a MacBook M3 on Sonoma so I will try to repro on my side with a node project.
Hey, thank you for your help. Here is the diagnostic id: 414254F5-98EE-4A97-84BC-433EEC56BF87/20240213162234 The last error I've posted I resolved by deleting a bunch of old unused images.
Either my allocated space was really full and docker didn't tell me about it, or there was some issue: After porting my data to the new M3 from intel mac I now noticed this:
although I remember having other settings, which the options seem to confirm:
I don't remember 67 GB of space resolving to only 20, is that because of that remark which is only shown in the settings? This might be a bit confusing to users. (Not sure if that is related to my issue or not).
Hey @nikita-fuchs,
I am also facing this issue while deploying docker image into AWS Elastic Beanstalk. So, do you have any idea about how to solve this issue in it ??
@dummyuser-123 I carefully purged all my unused/unneeded images and containers, and then it started working magically.
Looks like this issue was resolved, and without a reproducible scenario, not sure there's anything actionable here, so I'll go ahead and close.