app icon indicating copy to clipboard operation
app copied to clipboard

Fix that after a while, docker-desktop has deploys fail, with error "Docker is not responding. Maybe Docker is out of disk space? [...]"; seems to be memory issue

Open Venryx opened this issue 1 year ago • 1 comments

Likely related to WSL memory limits being reached.

Notes:

  • On my desktop, the WSL memory limit is 8gb.

Possibly related posts/issues

Related to memory:

  • Has tip that seems to help, once issue triggered / close to triggering: https://stackoverflow.com/a/66797264

Highly relevant, to rancher-desktop hang/slowness issue at least):

  • Core: https://github.com/microsoft/WSL/issues/8696
    • Spreadsheet of affected machine specs: https://docs.google.com/spreadsheets/d/10ML3KBdtINHYe8rSvat7JR_luVf1IeSammuAGVI983Q
  • Upstream (RD): https://github.com/rancher-sandbox/rancher-desktop/issues/6315
  • Upstream (RD): https://github.com/rancher-sandbox/rancher-desktop/issues/6057

Cases

  • 2024-05-19 (v-desktop): Docker-desktop showed memory usage of 8.01gb at time of failed deploy. (ie. at WSL memory limit)
  • 2024-05-23 (v-desktop): Issue happened; DD was not showing memory-usage yet, but did show an images list; pressed "Stop" on app-server, to free up a bit of memory; DD then showed memory usage of 8.01gb (ie. at WSL memory limit); Lens was then able to reconnect.

Toward resolution

Notes:

  • Shutting down Docker Desktop, then starting it again, seems to fix the issue temporarily. (that + killing the app-server pod in DD, appears to reduce memory usage by ~450mb)

Next time issue happens, try running in WSL/Ubuntu (in sudo bash): sync; echo 3 > /proc/sys/vm/drop_caches (reduced memory usage by ~2.3gb; unsure if actually helps with the deploy problem though, since it might be "illusory" savings if was just caching memory that would have been auto-dropped as needed anyway)

Venryx avatar May 19 '24 18:05 Venryx

Found a possible fix: (from: https://github.com/microsoft/WSL/issues/8725#issuecomment-1260627017)

[wsl]
kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all

[experimental]
autoMemoryReclaim=gradual

Except I replaced "gradual" with "dropCache" in the above. (note: "dropCache" is the default as of WSL2 v2.2.1)

Venryx avatar May 23 '24 23:05 Venryx