sysbox 0.6.1 does not support K8S 1.23.x fsGroup PodSecurityContext making all kind of non-host volumes useless

[FFock]

trafficstars

Install an Ubuntu image with a EmptyDir volume and a Longhorn volume on a Kubernetes 1.23.16 cluster created with Rancher 2.7.3 RKE1 on Ubuntu 20.04 (22.04 does not work with sysbox installer, see https://github.com/nestybox/sysbox/issues/567) and specify "fsGroup: 1000" in the pods securityContext as described here https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods

You will see that the empty dir volume as well as the longhorn volume will be mounted as:

drwxrwxrwx   2 nobody nogroup 4096 May 24 17:02 data
drwxrwsr-x   3 nobody nogroup 4096 May 24 15:39 data2

Even if you do not use the fsGroup security context, the volumes are mounted the same. When running the pod without sysbox runtime, everything works as exptected, i.e., with fsGroup 1000, the volumes are mounted at GID 1000 and without fsGroup they are mounted as group ID 0.

[FFock]

With RKE2, K8S 1.25.9 and Ubuntu 22.04, the Longhorn volume mount with fsGroup pod security context works, however mounting a secret with fsGroup set, still mounts the secret with nobody:nogroup.

[rodnymolina]

@FFock, thanks for reporting this one.

A couple of questions:

• Could you please obtain journalctl -u sysbox-mgr output in the k8s node where sysbox was installed?
• Could you obtain a findmnt within one of the sysbox pods where you observe the volume issue?

Thanks.

[FFock]

@rodnymolina, sure, I can provide the requested data. Can I sent it on a private channel?

[rodnymolina]

@FFock, sure, please ping on slack.

[joncbenderkh]

@FFock just an fyi, we ran into a similar issue ourselves, it was resolved by using a 0.6.2 prerelease. We've been running that version in test and prod for a couple of weeks now with no issues. We did make a clone of that specific version to use, however.

[ctalledo]

we ran into a similar issue ourselves, it was resolved by using a 0.6.2 prerelease. We've been running that version in test and prod for a couple of weeks now with no issues.

Thanks @joncbenderkh for chiming in.

FYI, to use sysbox-deploy-k8s with the Sysbox v0.6.2 pre-release, simply replace the image: line in the sysbox-install YAML with this one:

image: ghcr.io/nestybox/sysbox-deploy-k8s:v0.6.2-dev

We should be releasing Sysbox v0.6.2 later this week.