[BUG] All pods in pending state or evicted: `invalid capacity 0 on image filesystem`

[iwilltry42]

What did you do

k3d cluster create, then observe all pods being in pending state.

What did you expect to happen

All default pods coming up as usual.

Screenshots or terminal output

$ kubectl describe node
...
Events:
  Type     Reason                   Age                     From                   Message
  ----     ------                   ----                    ----                   -------
  Normal   Starting                 6m28s                   kube-proxy             
  Normal   NodeHasNoDiskPressure    6m30s (x2 over 6m30s)   kubelet                Node k3d-k3s-default-server-0 status is now: NodeHasNoDiskPressure
  Normal   Starting                 6m30s                   kubelet                Starting kubelet.
  Warning  InvalidDiskCapacity      6m30s                   kubelet                invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  6m30s                   kubelet                Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientMemory  6m30s (x2 over 6m30s)   kubelet                Node k3d-k3s-default-server-0 status is now: NodeHasSufficientMemory
  Normal   NodeHasSufficientPID     6m30s (x2 over 6m30s)   kubelet                Node k3d-k3s-default-server-0 status is now: NodeHasSufficientPID
  Normal   Synced                   6m25s                   cloud-node-controller  Node synced successfully
  Normal   NodeHasDiskPressure      6m20s                   kubelet                Node k3d-k3s-default-server-0 status is now: NodeHasDiskPressure
  Normal   NodeReady                6m20s                   kubelet                Node k3d-k3s-default-server-0 status is now: NodeReady
  Normal   RegisteredNode           6m18s                   node-controller        Node k3d-k3s-default-server-0 event: Registered Node k3d-k3s-default-server-0 in Controller
  Warning  EvictionThresholdMet     4m10s (x14 over 6m20s)  kubelet                Attempting to reclaim ephemeral-storage
  Warning  FreeDiskSpaceFailed      90s                     kubelet                failed to garbage collect required amount of images. Wanted to free 166224175104 bytes, but freed 0 bytes

First potentially related error in server logs:

E0222 05:48:02.074907      23 cri_stats_provider.go:452] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"

Which OS & Architecture

• Ubuntu & MacOS reported at least

Which version of k3d

• v5.4.7 and HEAD builds
• v5.4.6

K3s versions

• v1.25.6+k3s1
• v1.24.4+k3s1

Which version of docker

• 22.10.14
• 22.10.18