Running container always produces sandbox when using spack-built singularity

[pelahi]

Version of Singularity:

3.7.1
3.7.2
3.7.3
3.7.4
3.8.0 

Expected behavior

I expect to have singularity exec image.sif ${exe} to not produce a sandbox when the container has been built without the sandbox option.

Actual behavior

I have built singularity with spack and also a manual build. These spack/manual built singularity's (variety of versions) are compiled with squashfs 4.4 and go 1.15. When running singularity on a container image it always produces a sandbox. However, using a ubuntu 20.04 provided singularity (installed via apt-get) does not show this behaviour. An example would be using a simple mpi osu test.

# using spack built singularity
$ mpirun -np 2 ${mpiargs} singularity run osutests.sif ${osu_latency} 
INFO:    Converting SIF file to temporary sandbox...
INFO:    Converting SIF file to temporary sandbox...
WARNING: underlay of /etc/localtime required more than 50 (98) bind mounts
WARNING: underlay of /etc/localtime required more than 50 (98) bind mounts# OSU MPI Latency Test v5.3
# Size          Latency (us)
0                       0.53
1                       0.55
...

# using ubuntu 20.04 apt-get install singularity
$ mpirun -np 2 ${mpiargs} singularity run osutests.sif ${osu_latency} 
# Size          Latency (us)
0                       0.53
1                       0.55
...

Note the absence of converting the SIF file to a sandbox in the latter installed singularity. This is the expected behaviour.

For completeness the spack built versions were installed using

# example install for 3.7.4
$ spack install [email protected]~suid ^[email protected] ^[email protected]

Steps to reproduce this behavior

I have tried several different OSs and architectures with spack verison 0.16.x though most of the tests used 0.16.2-3517-e1d7275f92

What OS/distro are you running

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

# also used 
NAME="SLES"
VERSION="15-SP2"
VERSION_ID="15.2"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP2"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp2"

How did you install Singularity

As mentioned above I installed singularity from source with spack, from source and manual build and also deb. The deb is the only one producing the expected behaviour.

[DrDaveD]

I would guess that spack does not build singularity as root with the setuid-root helper. Using a .sif file directly is a privileged operation currently, because it has to mount a squashfs filesystem, so the only thing singularity can do to run the container is convert it to sandbox. In the future on kernels that support unprivileged fuse mounts singularity will probably be able to use squashfuse to mount it directly.

[DrDaveD]

This is related to #3612.

[kmuriki]

Singularity repo is now retired as the code base is now moved to Apptainer. We are closing all the old issues under the old Singularity repo. For further assistance please open a new issue under the new Apptainer repo. Thanks for your support.