aibrix icon indicating copy to clipboard operation
aibrix copied to clipboard

MountVolume.SetUp failed for volume "kube-api-access-wfmh2" no space left on device

Open Jeffwan opened this issue 10 months ago • 2 comments

🐛 Describe the bug

Image

 Warning  FailedMount  25s (x19 over 37m)  kubelet            (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-wfmh2" : write /var/lib/kubelet/pods/26af5e70-f9b9-4c12-a9de-dcf2d4e93689/volumes/kubernetes.io~projected/kube-api-access-wfmh2/..2025_02_18_00_11_37.99831085/namespace: no space left on device
Warning  FailedMount        2m1s (x11 over 22m)  kubelet             (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-tzrpb" : write /var/lib/kubelet/pods/a951d40b-518a-4b29-a3c9-36ef96b4f09c/volumes/kubernetes.io~projected/kube-api-access-tzrpb/..2025_02_17_23_18_11.2933250311/ca.crt: no space left on device

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       2.0G  1.2G  775M  61% /
devtmpfs         24G     0   24G   0% /dev
tmpfs            24G     0   24G   0% /dev/shm
tmpfs           9.5G  6.2M  9.5G   1% /run
tmpfs            24G  248K   24G   1% /etc/machine-id
tmpfs           256K     0  256K   0% /mnt/disks
efivarfs         56K   24K   27K  48% /sys/firmware/efi/efivars
tmpfs            24G     0   24G   0% /tmp
overlayfs        24G  248K   24G   1% /etc
/dev/nvme0n1p8   11M   24K   11M   1% /usr/share/oem
/dev/nvme0n1p1   95G   57G   38G  60% /mnt/stateful_partition
tmpfs           2.0M  224K  1.8M  11% /var/lib/cloud
none             24G   38M   24G   1% /dev/kube_shm
tmpfs            24G   12K   24G   1% /var/lib/kubelet/pki

If has enough room but failed

Steps to Reproduce

latest version

kubectl apply -f samples/kvcache/kvcache.yaml
kubectl apply -f samples/kvcache/deployment.yaml

Expected behavior

it should not through any errors

Environment

nightly & GCP

Jeffwan avatar Feb 18 '25 00:02 Jeffwan

look at this issue and notice the default CPU & Memory are wrong, it all use value 2 instead of 2Gi. Update the kubebuilder default value

Jeffwan avatar Feb 18 '25 01:02 Jeffwan

Image

Jeffwan avatar Feb 18 '25 06:02 Jeffwan