CentOS containers from library do not unpack in unprivileged use

[vanderwb]

Version of Singularity:

3.7.2 3.8.0

Expected behavior

I expect to unpack the CentOS 8 container pulled from the SyLabs library and start a shell using singularity shell library://centos:8.

Actual behavior

On our cluster, the unpacking step fails with a permissions error. It appears that the container is unpacked into its root directory and then write-permissions are changed such that the "dev" path cannot be created. This causes the whole process to fail. Other containers (e.g., the Ubuntu containers) and the CentOS containers from DockerHub work, so I'm guessing that there is an issue with the default library CentOS container. Still, perhaps Singularity and unsquashfs should be able to handle this case gracefully?

Steps to reproduce this behavior

At least on our cluster, we have installs with allow setuid = no, and this seems to be sufficient to trigger the problem. We have used both version 4.2 and 4.4 of SquashFS, and we have used both GPFS and local /tmp storage to unpack.

$ SINGULARITY_TMPDIR=/tmp singularity shell library://centos:8
INFO:    Using cached image
INFO:    Converting SIF file to temporary sandbox...
FATAL:   while extracting /glade/u/home/vanderwb/.singularity/cache/library/sha256.037a37caf464991e7877893fae0b3694a4b8f5970fd8232c5de8c86f89dec97e: root filesystem extraction failed: could not create /tmp/rootfs-797161043/root/dev: mkdir /tmp/rootfs-797161043/root/dev: permission denied

Here is a snippet of debug output:

DEBUG   [U=27236,P=132713] Init()                        sif image format detected
DEBUG   [U=27236,P=132713] extract()                     Excluding /dev directory during root filesystem extraction (non root user)
DEBUG   [U=27236,P=132713] extract()                     Trying unsquashfs options: [-no-xattrs -r]
DEBUG   [U=27236,P=132713] unsquashfsSandboxCmd()        Calling wrapped unsquashfs: singularity [-q exec --no-home --no-nv --no-rocm -C --no-init --writable -B /tmp/rootfs-664608621:/image -B /sbin/unsquashfs:/sbin/unsquashfs:ro -B /lib64/libpthread.so.0:/lib64/libpthread.so.0:ro -B /lib64/libm.so.6:/lib64/libm.so.6:ro -B /lib64/libz.so.1:/lib64/libz.so.1:ro -B /lib64/liblzma.so.5:/lib64/liblzma.so.5:ro -B /lib64/liblzo2.so.2:/lib64/liblzo2.so.2:ro -B /lib64/libgcc_s.so.1:/lib64/libgcc_s.so.1:ro -B /lib64/libc.so.6:/lib64/libc.so.6:ro -B /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2:ro /tmp/rootfs-664608621/tmp-rootfs-549107546 /sbin/unsquashfs -no-xattrs -r -d /image/root /image/archive-491876840 ^(.{0}[^d]|.{1}[^e]|.{2}[^v]|.{3}[^\x2f]).*$]
FATAL   [U=27236,P=132713] execStarter()                 while extracting /glade/u/home/vanderwb/.singularity/cache/library/sha256.037a37caf464991e7877893fae0b3694a4b8f5970fd8232c5de8c86f89dec97e: root filesystem extraction failed: could not create /tmp/rootfs-664608621/root/dev: mkdir /tmp/rootfs-664608621/root/dev: permission denied

Finally, here is what permissions look like after the failure:

$ ls -dl /tmp/rootfs-664608621/root
dr-xr-xr-x 17 vanderwb csgteam 540 Apr 22  2020 /tmp/rootfs-664608621/root

What OS/distro are you running

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

How did you install Singularity

From source.