docker
Docker desktop have no response and the generated log files consume disk space rapidly
Description
After a long run, sometimes docker desktop has no response, and it generates a lot of log files such as electron-2024-02-21-17.log.xx in C:\Users\admin\AppData\Local\Docker\log\host.
The latest log is like this:
2024-02-21T05:16:14.953Z info [HEARTBEAT] containersOpened
2024-02-21T05:16:14.954Z info [HTTP] 200: POST /usage
2024-02-21T05:16:15.026Z info [HTTP] 200: GET /ping
2024-02-21T05:16:15.032Z info [MENU] Rebuilding with licenseHasBeenAccepted: false and loginRestricted: false
2024-02-21T05:16:15.033Z info [MENU] Rebuilding with licenseHasBeenAccepted: false and loginRestricted: false
2024-02-21T05:16:15.034Z info [MENU] Rebuilding with licenseHasBeenAccepted: false and loginRestricted: false
2024-02-21T05:16:15.034Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.035Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.038Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.043Z info [HTTP] 200: GET /versions
2024-02-21T05:16:15.043Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.044Z info [HTTP] 200: GET /backend/state
2024-02-21T05:16:15.045Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.046Z info [HTTP] 200: GET /docker
2024-02-21T05:16:15.048Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.048Z info [HTTP] 200: GET /app/settings
2024-02-21T05:16:15.049Z info [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.050Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.051Z info [HTTP] 200: GET /features
2024-02-21T05:16:15.051Z info [HTTP] 200: GET /install/check
2024-02-21T05:16:15.052Z info [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.053Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.054Z info [HTTP] 200: GET /kubernetes
2024-02-21T05:16:15.055Z info [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.055Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.055Z info [HTTP] 200: GET /features
2024-02-21T05:16:15.057Z info [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.057Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.061Z info [HTTP] 200: GET /store
2024-02-21T05:16:15.075Z info [HTTP] 200: GET /store
2024-02-21T05:16:15.079Z info [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.079Z info [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.101Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.102Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.102Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.103Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.103Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.103Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.103Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.104Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.104Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.104Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.104Z error Unhandled uncaught exception : {}
2024-02-21T05:16:15.105Z error Unhandled uncaught exception : {}
......
Then it only has the same Unhandled uncaught exception in the later log file.
These log files take around 2.7 GB per hour, and they quickly run out of my disk!
Reproduce
run some containers and wait. recently i usually ran containers (ubuntu based) with a lot of memory usage.
Expected behavior
No response
docker version
Client:
Cloud integration: v1.0.35+desktop.10
Version: 25.0.3
API version: 1.44
Go version: go1.21.6
Git commit: 4debf41
Built: Tue Feb 6 21:13:02 2024
OS/Arch: windows/amd64
Context: default
Server: Docker Desktop 4.27.2 (137060)
Engine:
Version: 25.0.3
API version: 1.44 (minimum version 1.24)
Go version: go1.21.6
Git commit: f417435
Built: Tue Feb 6 21:14:25 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.28
GitCommit: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Version: 25.0.3
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1-desktop.4
Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe
compose: Docker Compose (Docker Inc.)
Version: v2.24.5-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe
debug: Get a shell into any image or container. (Docker Inc.)
Version: 0.0.24
Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: C:\Program Files\Docker\cli-plugins\docker-dev.exe
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.21
Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.4
Path: C:\Program Files\Docker\cli-plugins\docker-feedback.exe
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.0.0
Path: C:\Program Files\Docker\cli-plugins\docker-init.exe
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe
scout: Docker Scout (Docker Inc.)
Version: v1.4.1
Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe
Server:
Containers: 12
Running: 1
Paused: 0
Stopped: 11
Images: 21
Server Version: 25.0.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
Kernel Version: 5.15.133.1-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 15.49GiB
Name: docker-desktop
ID: b91cf323-d6dc-44b1-8280-56ee16b8cac2
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Registry Mirrors:
https://docker.mirrors.ustc.edu.cn/
https://mirror.ccs.tencentyun.com/
https://dockerproxy.com/
https://docker.nju.edu.cn/
https://docker.m.daocloud.io/
Live Restore Enabled: false
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile
Diagnostics ID
B3E4D14A-8D70-493F-8663-FB1918D23269/20240222081303
Additional Info
No response
I have experienced the same for some time. Its been very on and off so its been difficult to diagnose.
Literally yesterday i figured out what caused this issue in our environment. Not sure if its the same cause as in your scenario, but our issue came from our automation in starting/stopping of Docker Desktop. Do you automate starting/stopping of Docker Desktop?
In our automation we had a bug that started two instances of Docker Desktop in rapid succession. Which in some scenarios caused the dashboard to freeze and we got these errors in the logs as you mentioned above.
I have managed to recreate the issue with the following code. It only occurs in approximately 1 out of 5 times
Get-Process com.docker*| Stop-Process -Force -ErrorAction SilentlyContinue
Get-Process docker* | Stop-Process -Force -ErrorAction SilentlyContinue
Get-Process Docker* | Stop-Process -Force -ErrorAction SilentlyContinue
Start-Sleep 5
Start-Process -FilePath "$Env:ProgramFiles\Docker\Docker\Docker Desktop.exe"
Start-Process -FilePath "$Env:ProgramFiles\Docker\Docker\Docker Desktop.exe"
Edit: I have reproduced this on a fresh updated Win11 VM, manually installed Docker desktop and with no containers
I don't use any automation. My work process is: turn on my computer -> manually start Docker Desktop -> open WSL and locate to my project -> open vscode from WSL -> reopen in devcontainer -> kill Docker Desktop if it stuck (maybe OOM)
I have (had) the same issue.
The line error Unhandled uncaught exception : {} was being written to logs hundreds of times per second, adding ~18MB of log files every minute. This has apparently been going on for several weeks, and I ended up with 75GB of log files.
Disabling "Docker Desktop Launcher" in Windows task manager -> Startup seems to have fixed the issue.
I got the same issues as @Neorej on one of our build servers.
We added 96GB in 3 days. Here is the first file of hundreds:
electron-2024-03-01-07.log
Interestingly it started on the first of the month.
The electronxxx.log files have filled my hdd. docker was acting flaky on startup; tried running update with no success. finally manually downloaded installer and reran that to get docker to current version. its now starting fine. however, I have about 110gb of log files in \appdata\local\docker\log\host.
this is the ONLY (??) post I have found referencing these log files, so I am asking the question here: what files can I safely delete?
eg which of these can I delete electron-2024-04-13-06.log electron-2024-04-13-06.log.1 electron-2024-04-13-06.log.2 ... electron-2024-04-13-06.log.24 etc
fyi content of files is "2024-04-12T11:00:00.000Z error Unhandled uncaught exception : {}"
diagnostics id (I think) 3B8749F4-C761-45F4-AA4D-B383BD460D29/20240423205743
We have the same issues, an overflowing amount of electron logs have filled up the system drive one of our build servers, taking it down on several occasions now.
It is not constant, as it doesn't happen all the time according to our disk usage metrics, but after it starts it seems to continue until there is no disk space left.
The Docker Desktop version on it is 4.29.0 (145265), so not the same as the original poster.
Just had this issue on a Mac as well.
I had noticed no problems with Docker in weeks prior, but woke up to find my 500GB disk full of 384GB system data, which I managed to track down to docker logs. docker system prune and docker system prune -af didn't do anything to clear them up. I ended up deleting the folder manually.
Apparently this is not only a docker for-win issue.
My organization is also seeing this same behavior on Docker for Mac too.
I just checked and found 180GB of logs in ~/Library/Containers/com.docker.docker/Data/log/host/electron.*.log
024-06-09T21:15:00.509Z error Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }
2024-06-09T21:15:00.510Z error Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }
2024-06-09T21:15:00.510Z error Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }
2024-06-09T21:15:00.510Z error Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }
We use k8s for docker. The daemon and client are regularly crashing. It seems like the client could better handle when some thread starts throwing an exception like this.
Here are my example logs from when things go haywire:
Investigating the logs, the final "normal" log message before bouts of Unhandled uncaught exception is relating to /stats. This correlates with the docker daemon being unavailable from the command line.
My environment has heavy memory usage. Desktop: 4.30.0 (149282) Engine: 26.1.1 Kubernetes: v1.29.2
This only seems to have happened recently, I've been testing Kubernetes in WSL2-Docker in a VM (Yep, nested virtualization) and after updating this week, it seems the docker desktop spam is intent in filling my VM without an option to disable logging from the client.
160GB of wasted space in three days.
2024-06-17T14:44:49.246Z error Unhandled uncaught exception : { message: 'EPIPE: broken pipe, write', errno: -4047, syscall: 'write', code: 'EPIPE' } 2024-06-17T14:44:49.246Z info [SSE] Disconnecting from /kubernetes/events 2024-06-17T14:44:49.247Z info [SSE] Disconnecting from /registry/state 2024-06-17T14:44:49.248Z error Unhandled uncaught exception : { message: 'EPIPE: broken pipe, write', errno: -4047, syscall: 'write', code: 'EPIPE' } Mostly Epipe/broken pipe everywhere.
I use RDP to connect to my VMs, and I feel occasionally however windows handles sign outs / suspends likes to cause me issues with WSL2 + Docker Desktop + Kubernetes. I'll have to see if I can force the Host logs folder to be read only and not have docker desktop overwrite it
Yup. Same here. Filled my C-drive and I noticed this only after disk space was at 0 and Docker crashed and got notifications from my down detector. Turns out there were 264 gigabytes of electron log files on my nvme. WIN11E, Docker Desktop v4.31.1, engine 26.1.4.
edit: found out this issue yesterday, and after deleting the log files, there is now 1.2gb of logs files already.
Same issue on MAC M3. 247GB filled with log files.
docker version shows
Client:
Cloud integration: v1.0.35+desktop.13
Version: 26.1.1
API version: 1.45
Go version: go1.21.9
Git commit: 4cf5afa
Built: Tue Apr 30 11:44:56 2024
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.30.0 (149282)
Engine:
Version: 26.1.1
API version: 1.45 (minimum version 1.24)
Go version: go1.21.9
Git commit: ac2de55
Built: Tue Apr 30 11:48:04 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.31
GitCommit: e377cd56a71523140ca6ae87e30244719194a521
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
I can repro on macOS as well.
macOS Sonoma 14.5 Docker Desktop 4.32.0 (157355) Engine: 27.0.3
❯ sudo du -h -d1 /System/Volumes/Data/Users/patrick.nausha/Library/Containers/com.docker.docker/Data/log/host | sort -h
309G /System/Volumes/Data/Users/patrick.nausha/Library/Containers/com.docker.docker/Data/log/host
Several thousand electron files are in that directory.
❯ ls -l /System/Volumes/Data/Users/patrick.nausha/Library/Containers/com.docker.docker/Data/log/host | grep 'electron-' | wc -l
16255
The logs contain mostly (exclusively?) these lines which resemble what others in this issue see:
2024-07-10T17:52:16.871Z error Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }
Hi all, we're trying to address this issue internally. Could you please try out this build to see if the issue still persists?
Windows: https://desktop-stage.docker.com/win/main/amd64/159873/Docker%20Desktop%20Installer.exe
I see some Mac users here, these are Mac builds:
Mac M1: https://desktop-stage.docker.com/mac/main/arm64/159873/Docker.dmg Mac Intel: https://desktop-stage.docker.com/mac/main/amd64/159873/Docker.dmg
I'd like to know if one of you have a least reproduction case, or at least any symptom of malfunctioning of Docker Desktop in case this kind of log start to accumulate.
Sorry for any inconvenience.
I have the same problem. My docker desktop info: Current version: 4.30.0 (149282)
electron.log content: 2024-07-19T15:11:53.900Z error Unhandled uncaught exception : { message: 'EPIPE: broken pipe, write', errno: -4047, syscall: 'write', code: 'EPIPE' }
I'll be leaving my usual containers running in my RDP'd Win10 VM (nested virt on, WSL2 enabled, Ubuntu 24.04 default attached distro) with the latest version of Docker-Desktop (Engine v27.03, Kubernetes 1.29.2, Desktop V4.32) over the weekend to see what issues come up, if any.
I've not had the same massive log spam use up all my disk space in the past few weeks and I regularly update my docker desktop, but I do think a better option would be to force the electron backend to have a limit of 8GB of logfiles on rotation by default to avoid this entirely?
to have a limit of 8GB of logfiles on rotation by default to avoid this entirely?
Agreed, having a configurable limit with a sensible default ought to be implemented as a protection against this class of issue, in addition to fixing the particular instance.
Can reproduce this on mac Server: Docker Desktop 4.30.0 (149282) .
Had to delete more than 280GB of electron logs
Hi all, we're trying to address this issue internally. Could you please try out this build to see if the issue still persists?
Windows: desktop-stage.docker.com/win/main/amd64/159873/Docker%20Desktop%20Installer.exe
I see some Mac users here, these are Mac builds:
Mac M1: desktop-stage.docker.com/mac/main/arm64/159873/Docker.dmg Mac Intel: desktop-stage.docker.com/mac/main/amd64/159873/Docker.dmg
I'd like to know if one of you have a least reproduction case, or at least any symptom of malfunctioning of Docker Desktop in case this kind of log start to accumulate.
Sorry for any inconvenience.
Hi all,
I'd like to reiterate my request above. You could even try the latest release of Docker Desktop (4.33.0) https://www.docker.com/products/docker-desktop/ to see if the issue still persists.
Sorry for any inconvenience.
Just found this after troubleshooting a full disk issue, also due to many 20481kb electron logs filling the drive. Docker Desktop process was hanging also, required close in task manager then reopen to use. Docker Desktop 4.28.0 (139021) on win10.
Experiencing this issue now for the first time after updating to Docker Desktop 4.34 on my Windows 10 machine. Had to change my Microsoft pin and sign out of all applications that use Windows login to even log into the machine.
Also have been regularly experiencing errors in starting up Docker Engine with error messages like the following (not sure if they are related):
running engine: waiting for the VM setup to be ready: starting WSL engine: bootstrapping in the main distro: context canceled_
deploying WSL2 distributions ensuring main distro is deployed: checking if main distro is up to date: checking main distro bootstrap version: getting main distro bootstrap version: open \wsl$\docker-desktop\etc\wsl_bootstrap_version: The network name cannot be found. checking if isocache exists: CreateFile \wsl$\docker-desktop-data\isocache: The network name cannot be found.
running wsl distro proxy in Ubuntu-18.04 distro: exit status 1
Same on Mac M1 (Sonoma 14.6.1) version 4.34.0.
Electron logs consumed almost 220G in a day and a half, and all kinds of apps started failing because of maxed out disk.
I was able to recover by deleting "~/Library/Containers/com.docker.docker/Data/log/host/electron-*.log.*"
cross posted to https://github.com/docker/for-mac/issues/7348
