Cannot create pod with volume: Failed to eval symlink on : lstat 3: no such file or directory

[gabrielbull]

New kubernetes cluster running these versions:

• Kubernetes v1.30.6
• Ubuntu 22.04.5 LTS (GNU/Linux 6.8.0-1018-aws x86_64)
• crio version 1.30.4

Installed using the kubernetes file at https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-install.yaml which succeeded.

This error appears when trying to create a pod with a volume, works when the pod has no volume.

 Error: container create failed: time="2024-11-18T06:04:55Z" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: rootfs_init_linux.go:317: setting up ID-mapped mount on path 3 (likely means idmapped mounts are not supported on the filesystem at this path ()) caused: Failed to eval symlink on : lstat 3: no such file or directory"

Result of sudo systemctl status sysbox -n20

● sysbox.service - Sysbox container runtime
     Loaded: loaded (/lib/systemd/system/sysbox.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-11-18 06:02:29 UTC; 5min ago
       Docs: https://github.com/nestybox/sysbox
   Main PID: 6023 (sh)
      Tasks: 2 (limit: 38049)
     Memory: 396.0K
        CPU: 27ms
     CGroup: /system.slice/sysbox.service
             ├─6023 /bin/sh -c "/usr/bin/sysbox-runc --version && /usr/bin/sysbox-mgr --version && /usr/bin/sysbox-fs --version && /bin/sleep infinity"
             └─6045 /bin/sleep infinity

Nov 18 06:02:29 ip-172-31-38-51 systemd[1]: Started Sysbox container runtime.
Nov 18 06:02:29 ip-172-31-38-51 sh[6024]: sysbox-runc
Nov 18 06:02:29 ip-172-31-38-51 sh[6024]:         edition:         Community Edition (CE)
Nov 18 06:02:29 ip-172-31-38-51 sh[6024]:         version:         0.6.5
Nov 18 06:02:29 ip-172-31-38-51 sh[6024]:         commit:         1b440ff266841f3d2d296e664122a9e29ceb9fd8
Nov 18 06:02:29 ip-172-31-38-51 sh[6024]:         built at:         Sat Nov  9 06:09:34 UTC 2024
Nov 18 06:02:29 ip-172-31-38-51 sh[6024]:         built by:         Rodny Molina
Nov 18 06:02:29 ip-172-31-38-51 sh[6024]:         oci-specs:         1.1.0+dev
Nov 18 06:02:29 ip-172-31-38-51 sh[6031]: sysbox-mgr
Nov 18 06:02:29 ip-172-31-38-51 sh[6031]:         edition:         Community Edition (CE)
Nov 18 06:02:29 ip-172-31-38-51 sh[6031]:         version:         0.6.5
Nov 18 06:02:29 ip-172-31-38-51 sh[6031]:         commit:         1159d228eac8402efa63bd2cb18cdf9e404ea130
Nov 18 06:02:29 ip-172-31-38-51 sh[6031]:         built at:         Sat Nov  9 06:10:05 UTC 2024
Nov 18 06:02:29 ip-172-31-38-51 sh[6031]:         built by:         Rodny Molina
Nov 18 06:02:29 ip-172-31-38-51 sh[6038]: sysbox-fs
Nov 18 06:02:29 ip-172-31-38-51 sh[6038]:         edition:         Community Edition (CE)
Nov 18 06:02:29 ip-172-31-38-51 sh[6038]:         version:         0.6.5
Nov 18 06:02:29 ip-172-31-38-51 sh[6038]:         commit:         aeba775e52cc6385fa4807c594fc7ee164ad624c
Nov 18 06:02:29 ip-172-31-38-51 sh[6038]:         built at:         Sat Nov  9 06:10:01 UTC 2024
Nov 18 06:02:29 ip-172-31-38-51 sh[6038]:         built by:         Rodny Molina

[ctalledo]

Hi @gabrielbull, thanks for filing the issue.

The error setting up ID-mapped mount on path 3 suggests there's a problem parsing the path for the mount into the container.

Can you share the pod spec, particularly the parts that specify the mounts into the container?

Thanks.