Sysbox v0.6.2: container with too many layers fails to start

[ctalledo]

trafficstars

A container image with a large number of layers (> 64) fails to start with Sysbox v0.6.2, on my Ubuntu Kinetic host with kernel 5.19:

$ docker run -it --rm --runtime sysbox-runc codercom/oss-dogfood:testing
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:424: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:321: setting up ID-mapped mount on path l/UBOJSLP2TWYKIG7OSQIXTK24OU (likely means idmapped mounts are not supported on the filesystem at this path ()) caused: Failed to eval symlink on : lstat l: no such file or directory: unknown.

[ayazhafiz]

Depending on the storage driver being used, this may be due to a limitation that mount options cannot exceed 4096 bytes (https://lore.kernel.org/lkml/5670.1527244930@jrobl/T/). I'm not sure if that's the actual issue here, but may be a possibility, though I am sure you are already familiar with this.

[ctalledo]

Hi @ayazhafiz, yes it's likely that; I recall the problem was with the overlayfs mount in particular, where the image had so many layers that we exceeded the options limit when specifying the lowerdir option. Not sure how to fix it yet.

[ayazhafiz]

The way that Docker resolves this, and the way that we have done for a runc wrapper at my employer is to store all lowerdirs in the same directory (perhaps via linking entries in that directory to the real location of the lowerdir) and then calling mount in the context of that directory, and passing to mount relative paths. This way you can significantly shorten the names the lowerdir paths and shove many more of them in.

[ctalledo]

Thanks for that suggestion! I recall trying a similar approach back in August when I hit this issue, but there was some complication (don't recall which one) that prevented me from finalizing the solution, and then I got sidetracked by other issues. I would need to dig into it again to see what was going on ...