k0s
k0s copied to clipboard
k0s should move /var/lib/k0s/kubelet to /var/lib/kubelet
Before creating an issue, make sure you've checked the following:
- [X] You are running the latest released version of k0s
- [X] Make sure you've searched for existing issues, both open and closed
- [X] Make sure you've searched for PRs too, a fix might've been merged already
- [X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.
Version
1.23.6
Platform
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04 LTS
Release: 22.04
Codename: jammy
What happened?
aws-ebs-csi-driver out of the box failed to work with k0s.
Steps to reproduce
- Install k0s
- Install aws-ebs-csi-driver
- Install storageclass for aws-ebs-csi-driver
- Create pvc that will use the storageclass
Expected behavior
PVC creation via aws-ebs-csi-driver works
Actual behavior
PVC creation via aws-ebs-csi-driver fails
Screenshots and logs
Jun 13 05:53:05 nexus-nexus-01 k0s[4738]: time="2022-06-13 05:53:05" level=info msg="E0613 05:53:05.391274 4928 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/ebs.csi.aws.com^vol-0e2e37e9eecca4a86 podName: nodeName:}" failed. No retries permitted until 2022-06-13 05:54:09.391247943 +0000 UTC m=+1996.540517455 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "pvc-21851630-5134-4690-9807-91e57ef51902" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e2e37e9eecca4a86") pod "nexus-repo-nexus-repository-manager-5b5ff457c5-n5k9l" (UID: "753df29d-d7de-4d68-bd24-1538e1b1eafc") : applyFSGroup failed for vol vol-0e2e37e9eecca4a86: lstat /var/lib/k0s/kubelet/pods/753df29d-d7de-4d68-bd24-1538e1b1eafc/volumes/kubernetes.io~csi/pvc-21851630-5134-4690-9807-91e57ef51902/mount: no such file or directory" component=kubelet
Additional context
The issue is that aws-ebs-csi-driver
expects things in /var/lib/kubelet
not /var/lib/k0s/kubelet
.
Things tried:
- Symlinked
/var/lib/kubelet
to/var/lib/k0s/kubelet
- Symlinked
/var/lib/k0s/kubelet
to/var/lib/kubelet
- bind mount of
/var/lib/k0s/kubelet
to/var/lib/kubelet
- Modified the
aws-ebs-csi-driver
helm chart by hand to use/var/lib/k0s/kubelet
Results:
- same error
- same error
- same error
- works
In the process of tracking down this issue I ran across multiple previous k0s
issues and multiple CSI driver issues across GitHub projects that all point to many Kubernetes projects assume /var/lib/kubelet
. The ideal would be for it to always be configurable, but the reality is that people do assume. The ultimate issue is that k0s
is breaking compatibility with a lot of other projects by changing the directory from /var/lib/kubelet
to /var/lib/k0s/kubelet
.