Ben Cressey

> I think it's time that Bottlerocket have an official way to manage storage before kubelet starts up... or that there are parameters exposed that let us reconfigure kubelet to...

The symlink for `/var/lib/kubelet/pods` approach looks to be incompatible with at least the EBS and EFS CSI plugins, if not most others. These pods mount `/var/lib/kubelet` with the "Bidirectional" (`rshared`)...

What happens is that `kubelet` [checks CSI mounts for SELinux support](https://github.com/kubernetes/kubernetes/blob/4b024fc4eeb4a3eeb831e7fddec52b83d0b072df/pkg/volume/csi/csi_mounter.go#L273) which [inspects mount info for the seclabel option](https://github.com/kubernetes/kubernetes/blob/4b024fc4eeb4a3eeb831e7fddec52b83d0b072df/pkg/volume/util/hostutil/hostutil_linux.go#L237), using the mount info from [the first prefix match of the...

That leaves us with two requirements for CSI drivers to work: * `/var/lib/kubelet` must all reside on the same mount point (ignoring volume mounts, which come later) * `/var/lib/kubelet` must...

The only short-term fix to make CSI drivers work that I can see is to allow bootstrap containers to bind mount onto the storage directories, and have those mounts propagate...

I still need to work up an example of how to deal with "old" and "new" Bottlerocket approaches to mounting container storage in the same bootstrap container script.

I've been mulling over the long term fix for a while. Right now the tentative design I have in mind is to extend the user-data / first-boot configuration interface to...

@arnaldo2792 bootstrap containers would continue to be supported alongside actions. There'd need to be more host tools, like mdadm, lvm2, and devicemapper, but some of them might be required eventually...

@diranged - > Have you thought about just letting us mutate the kubelet config to our hearts content from within a bootstrap container? I have thought a lot about this....

An alternative that would make "config containers" much easier to use would be to just mount all of `/etc` into the container and give them the `super_t` label to let...