k0s should move /var/lib/k0s/kubelet to /var/lib/kubelet

[edgan]

Before creating an issue, make sure you've checked the following:

• [X] You are running the latest released version of k0s
• [X] Make sure you've searched for existing issues, both open and closed
• [X] Make sure you've searched for PRs too, a fix might've been merged already
• [X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Version

1.23.6

Platform

No LSB modules are available.

Distributor ID:	Ubuntu
Description:	Ubuntu 22.04 LTS
Release:	22.04
Codename:	jammy

What happened?

aws-ebs-csi-driver out of the box failed to work with k0s.

Steps to reproduce

1. Install k0s
2. Install aws-ebs-csi-driver
3. Install storageclass for aws-ebs-csi-driver
4. Create pvc that will use the storageclass

Expected behavior

PVC creation via aws-ebs-csi-driver works

Actual behavior

PVC creation via aws-ebs-csi-driver fails

Screenshots and logs

Jun 13 05:53:05 nexus-nexus-01 k0s[4738]: time="2022-06-13 05:53:05" level=info msg="E0613 05:53:05.391274 4928 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/ebs.csi.aws.com^vol-0e2e37e9eecca4a86 podName: nodeName:}" failed. No retries permitted until 2022-06-13 05:54:09.391247943 +0000 UTC m=+1996.540517455 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "pvc-21851630-5134-4690-9807-91e57ef51902" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e2e37e9eecca4a86") pod "nexus-repo-nexus-repository-manager-5b5ff457c5-n5k9l" (UID: "753df29d-d7de-4d68-bd24-1538e1b1eafc") : applyFSGroup failed for vol vol-0e2e37e9eecca4a86: lstat /var/lib/k0s/kubelet/pods/753df29d-d7de-4d68-bd24-1538e1b1eafc/volumes/kubernetes.io~csi/pvc-21851630-5134-4690-9807-91e57ef51902/mount: no such file or directory" component=kubelet

Additional context

The issue is that aws-ebs-csi-driver expects things in /var/lib/kubelet not /var/lib/k0s/kubelet.

Things tried:

1. Symlinked /var/lib/kubelet to /var/lib/k0s/kubelet
2. Symlinked /var/lib/k0s/kubelet to /var/lib/kubelet
3. bind mount of /var/lib/k0s/kubelet to /var/lib/kubelet
4. Modified the aws-ebs-csi-driver helm chart by hand to use /var/lib/k0s/kubelet

Results:

1. same error
2. same error
3. same error
4. works

In the process of tracking down this issue I ran across multiple previous k0s issues and multiple CSI driver issues across GitHub projects that all point to many Kubernetes projects assume /var/lib/kubelet. The ideal would be for it to always be configurable, but the reality is that people do assume. The ultimate issue is that k0s is breaking compatibility with a lot of other projects by changing the directory from /var/lib/kubelet to /var/lib/k0s/kubelet.