sysbox-fs - panic: runtime error: index out of range [0] with length 0

[matthewparkinsondes]

os=ubuntu 20.04.5, kernel=5.15.0-46-generic, docker=20.10.23, sysbox=0.5.2 EE

---

Attached logs show a panic in sysbox-fs.

panic_logs.txt

---

Steps to reproduce.

1. Within a system container sandbox (with Docker installed).
2. Obtain a Singularity .sif file.
3. docker run -v$PWD:/sif quay.io/singularity/singularity:v3.8.2 inspect /sif/test.sif

---

Summary of the relevant logs.

Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: panic: runtime error: index out of range [0] with length 0 Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: goroutine 5980 [running]: Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp.(*memParserIOvec).readProcessMem(0x1077528, 0x57509, 0xc000464ba0, 0x1, 0x1, 0xc000464b78, 0x1, 0x1, 0xc000464b70, 0x1, ...) Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp/memParserIOvec.go:99 +0x42f Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp.(*memParserIOvec).ReadSyscallBytesArgs(0x1077528, 0xc000057509, 0xc000606450, 0x1, 0x1, 0xc000604040, 0x2, 0x2, 0x0, 0x0) Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp/memParserIOvec.go:63 +0x1d7 Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp.(*syscallTracer).processSetxattr(0xc0000c6380, 0xc000476780, 0x12, 0xc75348, 0xc0002f0600, 0xba88ee, 0x8, 0x9384837acb9629bb, 0xc0002a26d8, 0x9384837acb9629bb) Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp/tracer.go:833 +0x1e2 Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp.(*syscallTracer).processSyscall(0xc0000c6380, 0xc000476780, 0xc000000012, 0xc00011e300, 0x40, 0xc0002a27b8, 0x5b5d4f, 0xc000184600) Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp/tracer.go:497 +0xc28 Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp.(*syscallTracer).process(0xc0000c6380, 0xc000476780, 0x12, 0xc00011e300, 0x40) Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp/tracer.go:439 +0xa7 Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: created by github.com/nestybox/sysbox-fs/seccomp.(*syscallTracer).connHandler Jan 31 16:14:33 athena207-testing sysbox-fs[346947]: github.com/nestybox/sysbox-fs/seccomp/tracer.go:420 +0x6e5 Jan 31 16:14:33 athena207-testing systemd[1]: sysbox-fs.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

---

[rodnymolina]

Thanks for reporting this one @matthewparkinsondes, will look into this.

[matthewparkinsondes]

Workaround.

sudo sed -i --follow-symlinks '/^ExecStart/ s/$/ --allow-trusted-xattr=false/' /lib/systemd/system/sysbox-mgr.service

[ctalledo]

sudo sed -i --follow-symlinks '/^ExecStart/ s/$/ --allow-trusted-xattr=false/' /lib/systemd/system/sysbox-mgr.service

Thanks @matthewparkinsondes; that means the bug is somewhere in the sysbox code that intercepts the xattr* syscalls inside the container. Fortunately the work-around is a good one in general, as setting --allow-trusted-xattr=false speeds up performance too.

[matthewparkinsondes]

Thanks, the performance speed up when setting --allow-trusted-xattr=false in the following scenario is greater than two orders of magnitude (> 30 mins vs ~5 secs).

docker run quay.io/singularity/singularity:v3.10.5 pull docker://tensorflow/tensorflow:latest

[ctalledo]

Thanks, the performance speed up when setting --allow-trusted-xattr=false in the following scenario is greater than two orders of magnitude (> 30 mins vs ~5 secs).

That's a lot (!) thanks for sharing.

[matthewparkinsondes]

Details from a recent Sysbox update.

• https://github.com/nestybox/sysbox/blob/master/docs/user-guide/configuration.md#speeding-up-sysbox-by-disallowing-trusted-overlay-xattributes--v062-or-earlier-

Starting with Sysbox v0.6.3, Sysbox starts with --allow-trusted-xattr=false by default. This improves performance (sometimes significantly) because Sysbox need not trap *xattr() syscalls.