When using sysbox 0.6.5, k8s secrets are mounted with owner `nobody nogroup`

[top-oai]

I'm running a pod in Kubernetes on a sysbox 0.6.5 runtime. In the Kubernetes spec, I've mounted a secret into a folder, and it's being mounted with owner nobody nogroup. This is no problem if we use the default k8s secret permissions of 0644, so all users can read, but if we choose to set e.g. defaultMode: 256 (0400 in oct) in the kubernets secret volume spec (which translates to 0400 where only the owner can read), then we're in trouble because root nor any other owner owns a nobody file. In this case, I hit a permission denied error. When sysbox is not installed, or if sysbox 0.6.4 or 0.6.3 is installed (I didn't observe this erroneous behavior before upgrading), then the owner is properly set to root.

I'm running on Linux kernel 5.15.0-1067-azure (it's an Azure AKS node).

I'm installing specifically using https://raw.githubusercontent.com/nestybox/sysbox/8834a2d9b8eabbfeb2a2181b8649684e14a619f1/sysbox-k8s-manifests/sysbox-install.yaml as my sysbox k8s install spec with no modifications.

I define my secrets in my k8s spec with:

  volumes:
  - name: github-deploy-key
    secret:
      defaultMode: 256
      secretName: github-deploy-key

and

    volumeMounts:
    - mountPath: /etc/secrets/ssh/github-deploy-key
      name: github-deploy-key
      readOnly: true

in this case I'm observing the same beahvior on both kubernetes versions 1.28.9 and 1.30.5.

[ctalledo]

Hi @top-oai, thanks for reporting the issue.

Can you do a findmnt within the container/pod, so I can see what filesystem is mounted on /etc/secrets/ssh/github-deploy-key?

Typically this problem is caused by Sysbox not being able to use idmapped mounts or shiftfs on the mounted filesystem, thus causing the files to show up with nobody:nogroup within the container.

[top-oai]

Can you do a findmnt within the container/pod, so I can see what filesystem is mounted on /etc/secrets/ssh/github-deploy-key?

Yes I certainly can:

TARGET                                      SOURCE              FSTYPE   OPTIONS
/                                           overlay             overlay  rw,relatime,lowerdir=3:7:8:9:10:11:12:...:47:
...
|-/etc/secrets/ssh/github-openai-deploy     tmpfs               tmpfs    ro,relatime,size=104857600k,inode64
...

(there of course is more output, so let me know if you think more would be helpful to see)

[ctalledo]

Thanks @top-oai.

|-/etc/secrets/ssh/github-openai-deploy tmpfs tmpfs ro,relatime,size=104857600k,inode64

That mount needs to have either shiftfs on it or be idmapped.

But ID-mapped mounts with tmpfs only work in kernel 6.3+, and you have kernel 5.15.

So the other solution is shiftfs, which must not be working, even though sysbox-deploy-k8s should have installed it (it's a kernel module).

Can you provide the output of journalctl -u sysbox-mgr on the K8s node? When sysbox-mgr starts, it checks if shiftfs is working on the node and logs this info.

[top-oai]

Sure here is the log output: https://gist.github.com/top-oai/8f5922dc2f1af36cc0aebd282b0a6b08

Notably I see this at the top:

Starting ...
Sysbox data root: /var/lib/sysbox
Shiftfs module found in kernel: yes
Shiftfs works properly: no
Shiftfs-on-overlayfs works properly: yes
ID-mapped mounts supported by kernel: yes
Overlayfs on ID-mapped mounts supported by kernel: no
Operating in system container mode.
Relaxed read-only mode disabled.
Inner container image preloading disabled.
Listening on /run/sysbox/sysmgr.sock
Ready ...

and I see this also (from https://github.com/nestybox/sysbox/blob/master/docs/user-guide/design.md#shiftfs-module):

$ sudo modprobe shiftfs
$ lsmod | grep shiftfs
shiftfs                28672  0

[ctalledo]

Thanks;

Shiftfs works properly: no

That's the problem; strange however, since the next line says:

Shiftfs-on-overlayfs works properly: yes

Thus, you must be hitting a bug we just found (and fixed) in the shiftfs checker. The fix is not yet released however, will be present in the upcoming v0.6.6 release (towards end of this week).

[top-oai]

Thus, you must be hitting a bug we just found (and fixed) in the shiftfs checker. The fix is not yet released however, will be present in the upcoming v0.6.6 release (towards end of this week).

Oh, wonderful! I look forward to trying it out, thank you!

[jojonium]

This issue appears to still be present in the latest version:

$ sysbox-mgr --version
sysbox-mgr
        edition:        Community Edition (CE)
        version:        0.6.6
        commit:         648c6e1d867e09f71bfb1303fa3e5af5cb6ebb54
        built at:       Sat Jan  4 00:00:59 UTC 2025
        built by:       Cesar Talledo
$ uname -r
5.15.0-1047-aws

sysbox-mgr logs:

Starting ...
Sysbox data root: /var/lib/sysbox
Shiftfs module found in kernel: yes
Shiftfs works properly: no
Shiftfs-on-overlayfs works properly: yes
ID-mapped mounts supported by kernel: yes
Overlayfs on ID-mapped mounts supported by kernel: no
Operating in system container mode.
Relaxed read-only mode disabled.
Inner container image preloading disabled.
Listening on /run/sysbox/sysmgr.sock
Ready ...

Inside a pod:

# ls -al /var/run/secrets/serviceaccount/token
lrwxrwxrwx 1 nobody nogroup 12 Jan 14 19:38 /var/run/secrets/serviceaccount/token -> ..data/token

[ctalledo]

Hi @jojonium, thanks for reporting.

What's the underlying filesystem on the K8s node at /var/lib/sysbox?

df -T /var/lib/sysbox

Is it ext4 or something else?

I ask because shiftfs works well on top of ext4 and overlayfs, but not always on top of other filesystems.

[jojonium]

Yes, the filesystem is ext4

[Peder2911]

I think I am hitting this as well, filesystem is ext4 on the node:

Token is not readable: