singularity
singularity copied to clipboard
sylabs
bind-mounting over autofs
SingularityCE version: 3.8.0 running on AARCH64 on Centos 7.9
We have a use case where we want to mount an autofs path inside singularity:
--bind /nfs/autofspath:/nfs/autofspath
This works fine and even if the path is not yet mounted on the host, it becomes available inside the container as requested.
However, we also want to bind-mount a different pat to a subdir of the above:
--bind /other-path:/nfs/autofspath/subdir1/subdir2
This works fine initially, however after a while of not using it, the host unmounts the /nfs/autofspath. When we try to access the /nfs/autofspath/subdir1/subdir2 path our 2nd bind-mount is lost, and we instead get the autofs mounted path.
If we use a bind-mount on the host to do the same thing, it persists and we do not have this issue.
This isn't really something that we can easily address, to the best of my knowledge.
The automounter is running in the host root mount namespace and is not seeing/considering the additional bind mount under /nfs/autofspath inside the container's mount namespace as a reason to hold onto the autofs mount.
We can't make singularity perform the nested bind on the host in order to hold open a file descriptor there, as this would mean container execution modifies the host filesystem view.
I'm afraid that it's likely that you will need to restructure how data is passed into, and accessed in the container.
It seems to work as expected if I flock the /nfs/autofspath path before starting singularity. I think this is similar to a previous autofs work-around for older kernels. Is there a way (or could there be an option) to have Singularity flock paths on the host before starting?
Hmm, perhaps I'm misunderstanding exactly what you are doing / what order things are happening here. It would be good to have a complete example showing exactly what mounts you have from mount, the exact and full command line used to run singularity, and --debug output.
The old autofs bug path operations now happen automatically, as singularity examines mount points to see if they are autofs. You should be able to see this happening if you run with singularity --debug .... It should report it's identifying them, and opening an fd.
Please also try to replicate on the latest version 3.9.5, as 3.8.0 is no supported and will not receive any fix. Thanks.
Here is the log using 3.9.5 (the issue is reproducible). Initially the /release/arm/subdir will contain the expected overlay data after a while /release becomes empty then querying /release/arm/subdir will give the underlying data instead of the overlay
FYI, I have moved to a different solution that doesn't require bind-mounting over autofs, though I would still like to be able to do this in the future if the need arises again:
singularity --debug exec --cleanenv --bind /release --bind /tools/subdir:/release/arm/subdir images/centos-7.9.compute.sif bash
DEBUG [U=5951,P=65350] persistentPreRun() Singularity version: 3.9.5-1.el7
DEBUG [U=5951,P=65350] persistentPreRun() Parsing configuration file /etc/singularity/singularity.conf
DEBUG [U=5951,P=65350] handleConfDir() /home/user/.singularity already exists. Not creating.
DEBUG [U=5951,P=65350] execStarter() Saving umask 0022 for propagation into container
DEBUG [U=5951,P=65350] execStarter() Checking for encrypted system partition
DEBUG [U=5951,P=65350] Init() Image format detection
DEBUG [U=5951,P=65350] Init() Check for sandbox image format
DEBUG [U=5951,P=65350] Init() sandbox format initializer returned: not a directory image
DEBUG [U=5951,P=65350] Init() Check for sif image format
DEBUG [U=5951,P=65350] Init() sif image format detected
DEBUG [U=5951,P=65350] SetContainerEnv() Forwarding TERM environment variable
VERBOSE [U=5951,P=65350] SetContainerEnv() Setting HOME=/home/user
VERBOSE [U=5951,P=65350] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=5951,P=65350] init() Use starter binary /usr/libexec/singularity/bin/starter-suid
DEBUG [U=5951,P=65350] Exec() Setting GOGC=off for starter
VERBOSE [U=0,P=65350] print() Set messagelevel to: 5
VERBOSE [U=0,P=65350] init() Starter initialization
DEBUG [U=0,P=65350] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=65350] load_overlay_module() Overlay seems supported by the kernel
VERBOSE [U=0,P=65350] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=65350] priv_drop() Drop root privileges
DEBUG [U=5951,P=65350] read_engine_config() Read engine configuration
DEBUG [U=5951,P=65350] cleanenv() Clearing env var LOAD_OVERLAY_MODULE=1
DEBUG [U=5951,P=65350] cleanenv() Keeping env var SINGULARITY_MESSAGELEVEL=5
DEBUG [U=5951,P=65350] cleanenv() Clearing env var ENGINE_CONFIG1={"engineName":"singularity","containerID":"","engineConfig":{"jsonConfig":{"bindpath":[{"source":"/release","destination":"/release","options":null},{"source":"/tools/subdir","destination":"/release/arm/subdir","options":null}],"unixSocketPair":[0,0],"image":"/home/user/git/singularity_cc/aarch64/centos-7.9.compute.sif","homedir":"/home/user","homeDest":"/home/user","network":"bridge","cwd":"/home/user","configurationFile":"/etc/singularity/si(TRUNCATED...)DEBUG [U=5951,P=65350] cleanenv() Clearing env var ENGINE_CONFIG_CHUNKS=1
DEBUG [U=5951,P=65350] cleanenv() Keeping env var GOGC=off
DEBUG [U=5951,P=65350] init() Wait completion of stage1
VERBOSE [U=5951,P=65361] priv_drop() Drop root privileges permanently
DEBUG [U=5951,P=65361] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=5951,P=65361] init() Spawn stage 1
DEBUG [U=5951,P=65361] startup() singularity runtime engine selected
VERBOSE [U=5951,P=65361] startup() Execute stage 1
DEBUG [U=5951,P=65361] StageOne() Entering stage 1
DEBUG [U=5951,P=65361] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=5951,P=65361] prepareAutofs() Found "/tools" as autofs mount point
DEBUG [U=5951,P=65361] prepareAutofs() Found "/release" as autofs mount point
DEBUG [U=5951,P=65361] prepareAutofs() Found "/home" as autofs mount point
DEBUG [U=5951,P=65361] keepAutofsMount() Open file descriptor for /release
DEBUG [U=5951,P=65361] keepAutofsMount() Open file descriptor for /tools/subdir
DEBUG [U=5951,P=65361] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=5951,P=65361] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=5951,P=65361] keepAutofsMount() Open file descriptor for /home/user
DEBUG [U=5951,P=65361] keepAutofsMount() Open file descriptor for /home/user
DEBUG [U=5951,P=65361] Init() Image format detection
DEBUG [U=5951,P=65361] Init() Check for sandbox image format
DEBUG [U=5951,P=65361] Init() sandbox format initializer returned: not a directory image
DEBUG [U=5951,P=65361] Init() Check for sif image format
DEBUG [U=5951,P=65361] Init() sif image format detected
DEBUG [U=5951,P=65361] setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=5951,P=65361] setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=5951,P=65350] wait_child() stage 1 exited with status 0
DEBUG [U=5951,P=65350] cleanup_fd() Setting FD_CLOEXEC on starter fd 3
DEBUG [U=5951,P=65350] cleanup_fd() Close file descriptor 4
DEBUG [U=5951,P=65350] cleanup_fd() Close file descriptor 5
DEBUG [U=5951,P=65350] cleanup_fd() Close file descriptor 6
DEBUG [U=5951,P=65350] cleanup_fd() Close file descriptor 7
DEBUG [U=5951,P=65350] cleanup_fd() Close file descriptor 8
DEBUG [U=5951,P=65350] cleanup_fd() Setting FD_CLOEXEC on starter fd 9
DEBUG [U=5951,P=65350] cleanup_fd() Setting FD_CLOEXEC on starter fd 10
DEBUG [U=5951,P=65350] cleanup_fd() Setting FD_CLOEXEC on starter fd 11
DEBUG [U=5951,P=65350] cleanup_fd() Setting FD_CLOEXEC on starter fd 12
DEBUG [U=5951,P=65350] init() Set child signal mask
DEBUG [U=5951,P=65350] init() Create socketpair for master communication channel
DEBUG [U=5951,P=65350] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=5951,P=65350] priv_escalate() Get root privileges
VERBOSE [U=0,P=65350] priv_escalate() Change filesystem uid to 5951
VERBOSE [U=0,P=65350] init() Spawn master process
DEBUG [U=0,P=65367] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=65367] create_namespace() Create mount namespace
VERBOSE [U=0,P=65350] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=65350] enter_namespace() Opening namespace file ns/mnt
DEBUG [U=0,P=65350] set_master_privileges() Set master privileges
DEBUG [U=0,P=65350] apply_privileges() Effective capabilities: 0x00000000000000c0
DEBUG [U=0,P=65350] apply_privileges() Permitted capabilities: 0x000001ffffffffff
DEBUG [U=0,P=65350] apply_privileges() Bounding capabilities: 0x000001ffffffffff
DEBUG [U=0,P=65350] apply_privileges() Inheritable capabilities: 0x000001ffffffffff
VERBOSE [U=0,P=65367] create_namespace() Create mount namespace
DEBUG [U=0,P=65350] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=65350] apply_privileges() Set user ID to 5951
DEBUG [U=0,P=65368] set_rpc_privileges() Set RPC privileges
DEBUG [U=0,P=65368] apply_privileges() Effective capabilities: 0x0000000000200000
DEBUG [U=0,P=65368] apply_privileges() Permitted capabilities: 0x000001ffffffffff
DEBUG [U=0,P=65368] apply_privileges() Bounding capabilities: 0x0000000008204000
DEBUG [U=0,P=65368] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=65368] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=65368] apply_privileges() Set user ID to 5951
DEBUG [U=5951,P=65368] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=5951,P=65368] init() Spawn RPC server
DEBUG [U=5951,P=65350] startup() singularity runtime engine selected
VERBOSE [U=5951,P=65350] startup() Execute master process
DEBUG [U=5951,P=65368] startup() singularity runtime engine selected
VERBOSE [U=5951,P=65368] startup() Serve RPC requests
DEBUG [U=5951,P=65350] setupSessionLayout() Using Layer system: overlay
DEBUG [U=5951,P=65350] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=5951,P=65350] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=5951,P=65350] addRootfsMount() Image type is 4096
DEBUG [U=5951,P=65350] addRootfsMount() Mounting block [squashfs] image: /home/user/git/singularity_cc/aarch64/centos-7.9.compute.sif
DEBUG [U=5951,P=65350] addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=5951,P=65350] addKernelMount() Adding proc to mount list
VERBOSE [U=5951,P=65350] addKernelMount() Default mount: /proc:/proc
DEBUG [U=5951,P=65350] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=5951,P=65350] addKernelMount() Adding sysfs to mount list
VERBOSE [U=5951,P=65350] addKernelMount() Default mount: /sys:/sys
DEBUG [U=5951,P=65350] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=5951,P=65350] addDevMount() Adding dev to mount list
VERBOSE [U=5951,P=65350] addDevMount() Default mount: /dev:/dev
DEBUG [U=5951,P=65350] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=5951,P=65350] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=5951,P=65350] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=5951,P=65350] addHomeStagingDir() Staging home directory (/home/user) at /var/singularity/mnt/session/home/user
DEBUG [U=5951,P=65350] addHomeMount() Adding home directory mount [/var/singularity/mnt/session/home/user:/home/user] to list using layer: overlay
DEBUG [U=5951,P=65350] addUserbindsMount() Adding /release to mount list
DEBUG [U=5951,P=65350] addUserbindsMount() Adding /tools/subdir to mount list
DEBUG [U=5951,P=65350] addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=5951,P=65350] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=5951,P=65350] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=5951,P=65350] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=5951,P=65350] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=5951,P=65350] addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=5951,P=65350] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=5951,P=65350] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=5951,P=65350] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=5951,P=65350] create() Mount all
DEBUG [U=5951,P=65350] mountGeneric() Mounting tmpfs to /var/singularity/mnt/session
DEBUG [U=5951,P=65350] mountGeneric() Mounting tmpfs to /var/singularity/mnt/session
DEBUG [U=5951,P=65350] mountImage() Mounting loop device /dev/loop0 to /var/singularity/mnt/session/rootfs of type squashfs
DEBUG [U=5951,P=65350] mountGeneric() Mounting overlay to /var/singularity/mnt/session/final
DEBUG [U=5951,P=65350] mountGeneric() Remounting /var/singularity/mnt/session/final
DEBUG [U=5951,P=65350] setPropagationMount() Set RPC mount propagation flag to SLAVE
VERBOSE [U=5951,P=65350] Passwd() Checking for template passwd file: /var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=5951,P=65350] Passwd() Creating passwd content
VERBOSE [U=5951,P=65350] Passwd() Creating template passwd file and appending user data: /var/singularity/mnt/session/rootfs/etc/passwd
DEBUG [U=5951,P=65350] addIdentityMount() Adding /etc/passwd to mount list
VERBOSE [U=5951,P=65350] addIdentityMount() Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=5951,P=65350] Group() Checking for template group file: /var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=5951,P=65350] Group() Creating group content
DEBUG [U=5951,P=65350] addIdentityMount() Adding /etc/group to mount list
VERBOSE [U=5951,P=65350] addIdentityMount() Default mount: /etc/group:/etc/group
DEBUG [U=5951,P=65350] mountGeneric() Mounting /dev to /var/singularity/mnt/session/final/dev
DEBUG [U=5951,P=65350] mountGeneric() Mounting /etc/localtime to /var/singularity/mnt/session/final/usr/share/zoneinfo/UTC
DEBUG [U=5951,P=65350] mountGeneric() Remounting /var/singularity/mnt/session/final/usr/share/zoneinfo/UTC
DEBUG [U=5951,P=65350] mountGeneric() Mounting /etc/hosts to /var/singularity/mnt/session/final/etc/hosts
DEBUG [U=5951,P=65350] mountGeneric() Remounting /var/singularity/mnt/session/final/etc/hosts
DEBUG [U=5951,P=65350] mountGeneric() Mounting /proc to /var/singularity/mnt/session/final/proc
DEBUG [U=5951,P=65350] mountGeneric() Remounting /var/singularity/mnt/session/final/proc
DEBUG [U=5951,P=65350] mountGeneric() Mounting sysfs to /var/singularity/mnt/session/final/sys
DEBUG [U=5951,P=65350] mountGeneric() Mounting /home/user to /var/singularity/mnt/session/home/user
DEBUG [U=5951,P=65350] mountGeneric() Remounting /var/singularity/mnt/session/home/user
DEBUG [U=5951,P=65350] mountGeneric() Mounting /var/singularity/mnt/session/home/user to /var/singularity/mnt/session/final/home/user
DEBUG [U=5951,P=65350] mountGeneric() Mounting /tmp to /var/singularity/mnt/session/final/tmp
DEBUG [U=5951,P=65350] mountGeneric() Remounting /var/singularity/mnt/session/final/tmp
DEBUG [U=5951,P=65350] mountGeneric() Mounting /var/tmp to /var/singularity/mnt/session/final/var/tmp
DEBUG [U=5951,P=65350] mountGeneric() Remounting /var/singularity/mnt/session/final/var/tmp
DEBUG [U=5951,P=65350] mountGeneric() Mounting /var/singularity/mnt/session/etc/resolv.conf to /var/singularity/mnt/session/final/etc/resolv.conf
DEBUG [U=5951,P=65350] mountGeneric() Mounting /var/singularity/mnt/session/etc/passwd to /var/singularity/mnt/session/final/etc/passwd
DEBUG [U=5951,P=65350] mountGeneric() Mounting /var/singularity/mnt/session/etc/group to /var/singularity/mnt/session/final/etc/group
DEBUG [U=5951,P=65350] mountGeneric() Mounting /release to /var/singularity/mnt/session/final/release
DEBUG [U=5951,P=65350] mountGeneric() Remounting /var/singularity/mnt/session/final/release
DEBUG [U=5951,P=65350] mountGeneric() Mounting /tools/subdir to /var/singularity/mnt/session/final/release/arm/subdir
DEBUG [U=5951,P=65350] mountGeneric() Remounting /var/singularity/mnt/session/final/release/arm/subdir
DEBUG [U=5951,P=65350] addCwdMount() Using /home/user as current working directory
VERBOSE [U=5951,P=65350] addCwdMount() /home/user found within container
DEBUG [U=5951,P=65350] create() Chroot into /var/singularity/mnt/session/final
DEBUG [U=5951,P=65368] Chroot() Hold reference to host / directory
DEBUG [U=5951,P=65368] Chroot() Called pivot_root on /var/singularity/mnt/session/final
DEBUG [U=5951,P=65368] Chroot() Change current directory to host / directory
DEBUG [U=5951,P=65368] Chroot() Apply slave mount propagation for host / directory
DEBUG [U=5951,P=65368] Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=5951,P=65368] Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=5951,P=65350] create() Chdir into / to avoid errors
VERBOSE [U=0,P=65367] wait_child() rpc server exited with status 0
DEBUG [U=0,P=65367] init() Set container privileges
DEBUG [U=0,P=65367] apply_privileges() Effective capabilities: 0x0000000000000000
DEBUG [U=0,P=65367] apply_privileges() Permitted capabilities: 0x0000000000000000
DEBUG [U=0,P=65367] apply_privileges() Bounding capabilities: 0x0000000000000000
DEBUG [U=0,P=65367] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=65367] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=65367] apply_privileges() Set user ID to 5951
DEBUG [U=5951,P=65367] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=5951,P=65367] startup() singularity runtime engine selected
VERBOSE [U=5951,P=65367] startup() Execute stage 2
DEBUG [U=5951,P=65367] StageTwo() Entering stage 2
DEBUG [U=5951,P=65367] StartProcess() Setting umask in container to 0022
DEBUG [U=5951,P=65367] sylogBuiltin() Sourcing /.singularity.d/env/01-base.sh
DEBUG [U=5951,P=65367] sylogBuiltin() Sourcing /.singularity.d/env/10-docker2singularity.sh
DEBUG [U=5951,P=65367] sylogBuiltin() Sourcing /.singularity.d/env/90-environment.sh
DEBUG [U=5951,P=65367] sylogBuiltin() Sourcing /.singularity.d/env/94-appsbase.sh
DEBUG [U=5951,P=65367] sylogBuiltin() Sourcing /.singularity.d/env/95-apps.sh
DEBUG [U=5951,P=65367] sylogBuiltin() Sourcing /.singularity.d/env/99-base.sh
DEBUG [U=5951,P=65367] sylogBuiltin() Sourcing /.singularity.d/env/99-runtimevars.sh
DEBUG [U=5951,P=65367] sylogBuiltin() Running action command exec
DEBUG [U=5951,P=65350] PostStartProcess() Post start process