singularity
singularity copied to clipboard
Published
20 hours ago •
apptainer
apptainer
Non-standard home folder location: FATAL container creation failed: failed to add as session directory: path . is not an absolute path
Issue
Containers run normally for users whose home is /home/USERNAME but not things like /home/DOMAIN/USERNAME
Both directory locations are NFS mounts and are working.
Version of Singularity:
$ singularity --version
singularity version 3.8.2+92-gf6dfd98
Expected behavior
Containers should run.
Actual behavior
containers fails with FATAL [U=1065066,P=15042]Master() container creation failed: failed to add as session directory: path . is not an absolute path
Steps to reproduce this behavior
$ singularity --debug run shub://GodloveD/lolcow
DEBUG [U=1065066,P=15042]persistentPreRun() Singularity version: 3.8.2+92-gf6dfd98
...
Process
...
DEBUG [U=1065066,P=15042]persistentPreRun() Parsing configuration file /resource/apps/singularity/installation/etc/singularity/singularity.conf
DEBUG [U=1065066,P=15042]handleConfDir() /home/REGISTRY/MY_USER/.singularity already exists. Not creating.
DEBUG [U=1065066,P=15042]setValue() Updated flag 'tmpdir' value to: /scratch/MY_USER/singularity/tmp
DEBUG [U=1065066,P=15042]GetManifest() shub request: https://singularity-hub.org/api/container/GodloveD/lolcow:latest
DEBUG [U=1065066,P=15042]GetManifest() 200 OK response received, beginning manifest download
DEBUG [U=1065066,P=15042]GetManifest() manifest image name: GodloveD/lolcow
INFO [U=1065066,P=15042]pull() Use cached image
DEBUG [U=1065066,P=15042]execStarter() Saving umask 0022 for propagation into container
VERBOSE [U=1065066,P=15042]execStarter() Unprivileged installation: using user namespace
DEBUG [U=1065066,P=15042]execStarter() Checking for encrypted system partition
DEBUG [U=1065066,P=15042]Init() Image format detection
DEBUG [U=1065066,P=15042]Init() Check for sandbox image format
DEBUG [U=1065066,P=15042]Init() sandbox format initializer returned: not a directory image
DEBUG [U=1065066,P=15042]Init() Check for sif image format
DEBUG [U=1065066,P=15042]Init() sif format initializer returned: SIF magic not found
DEBUG [U=1065066,P=15042]Init() Check for squashfs image format
DEBUG [U=1065066,P=15042]Init() squashfs image format detected
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding MANPATH environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding XDG_SESSION_ID environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding HOSTNAME environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding __LMOD_REF_COUNT_MODULEPATH environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LMOD_ROOT environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding HISTSIZE environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding SSH_CLIENT environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding MODULEPATH_ROOT environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LMOD_PKG environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding QTDIR environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding QTINC environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LMOD_VERSION environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding SSH_TTY environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding __LMOD_REF_COUNT_LOADEDMODULES environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding __LMOD_REF_COUNT_CMAKE_PREFIX_PATH environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding POWERLINE_COMMAND environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding EBVERSIONSINGULARITY environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding QT_GRAPHICSSYSTEM_CHECKED environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding http_proxy environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding USER environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding HISTFILESIZE environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LS_COLORS environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LMOD_sys environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding __LMOD_REF_COUNT__LMFILES_ environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding ftp_proxy environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding FTP_PROXY environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding COBBLER_SERVER environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding SCRDIR environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding TMUX environment variable
VERBOSE [U=1065066,P=15042]SetContainerEnv() Not forwarding SINGULARITY_TMPDIR environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding ALL_PROXY environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding _ModuleTable001_ environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding MAIL environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding _LMFILES_ environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding MODULEPATH environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding _ModuleTable_Sz_ environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LOADEDMODULES environment variable
VERBOSE [U=1065066,P=15042]SetContainerEnv() Not forwarding SINGULARITY_CACHEDIR environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding TMUX_PANE environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LMOD_CMD environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding https_proxy environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding HTTPS_PROXY environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding HISTCONTROL environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding __LMOD_REF_COUNT_PATH environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding _ModuleTable002_ environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding no_proxy environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding HTTP_PROXY environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding BASH_ENV environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding QTLIB environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding SSH_CONNECTION environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding MODULESHOME environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LMOD_SETTARG_FULL_SUPPORT environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LESSOPEN environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding PROMPT_COMMAND environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding EBROOTSINGULARITY environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding CMAKE_PREFIX_PATH environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding DISPLAY environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding LMOD_DIR environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding SCRATCH environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding MOD_PATH environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding BASH_FUNC_module() environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding BASH_FUNC_ml() environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding _ environment variable
DEBUG [U=1065066,P=15042]SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=1065066,P=15042]SetContainerEnv() Setting HOME=/home/REGISTRY/MY_USER
VERBOSE [U=1065066,P=15042]SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=1065066,P=15042]findFromConfig() Using "unsquashfs" at "/usr/sbin/unsquashfs" (from singularity.conf)
VERBOSE [U=1065066,P=15042]execStarter() User namespace requested, convert image /scratch/MY_USER/singularity/cache/cache/shub/a59d8de3121579fe9c95ab8af0297c2e3aefd827 to sandbox
INFO [U=1065066,P=15042]execStarter() Converting SIF file to temporary sandbox...
DEBUG [U=1065066,P=15042]Init() Image format detection
DEBUG [U=1065066,P=15042]Init() Check for sandbox image format
DEBUG [U=1065066,P=15042]Init() sandbox format initializer returned: not a directory image
DEBUG [U=1065066,P=15042]Init() Check for sif image format
DEBUG [U=1065066,P=15042]Init() sif format initializer returned: SIF magic not found
DEBUG [U=1065066,P=15042]Init() Check for squashfs image format
DEBUG [U=1065066,P=15042]Init() squashfs image format detected
DEBUG [U=1065066,P=15042]findFromConfig() Using "unsquashfs" at "/usr/sbin/unsquashfs" (from singularity.conf)
DEBUG [U=1065066,P=15042]extract() Excluding /dev directory during root filesystem extraction (non root user)
DEBUG [U=1065066,P=15042]extract() Trying unsquashfs options: [-no-xattrs -r]
DEBUG [U=1065066,P=15042]unsquashfsSandboxCmd() Calling wrapped unsquashfs: singularity [-q exec --no-home --no-nv --no-rocm -C --no-init --writable -B /scratch/MY_USER/singularity/tmp/rootfs-503788436:/image -B /usr/sbin/unsquashfs:/usr/sbin/unsquashfs:ro -B /usr/lib64/libpthread.so.0:/usr/lib64/libpthread.so.0:ro -B /usr/lib64/libm.so.6:/usr/lib64/libm.so.6:ro -B /usr/lib64/libz.so.1:/usr/lib64/libz.so.1:ro -B /usr/lib64/liblzma.so.5:/usr/lib64/liblzma.so.5:ro -B /usr/lib64/liblzo2.so.2:/usr/lib64/liblzo2.so.2:ro -B /usr/lib64/libgcc_s.so.1:/usr/lib64/libgcc_s.so.1:ro -B /usr/lib64/libc.so.6:/usr/lib64/libc.so.6:ro -B /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2:ro /scratch/MY_USER/singularity/tmp/rootfs-503788436/tmp-rootfs-966457869 /usr/sbin/unsquashfs -no-xattrs -r -d /image/root /image/archive-865596899 ^(.{0}[^d]|.{1}[^e]|.{2}[^v]|.{3}[^\x2f]).*$]
DEBUG [U=1065066,P=15042]init() Use starter binary /resource/apps/singularity/installation/libexec/singularity/bin/starter
VERBOSE [U=1065066,P=15042] print() Set messagelevel to: 5
VERBOSE [U=1065066,P=15042] init() Starter initialization
VERBOSE [U=1065066,P=15042] is_suid() Check if we are running as setuid
DEBUG [U=1065066,P=15042] read_engine_config() Read engine configuration
DEBUG [U=1065066,P=15042] init() Wait completion of stage1
DEBUG [U=1065066,P=15499] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1065066,P=15499] init() Spawn stage 1
DEBUG [U=1065066,P=15499]startup() singularity runtime engine selected
VERBOSE [U=1065066,P=15499]startup() Execute stage 1
DEBUG [U=1065066,P=15499]StageOne() Entering stage 1
DEBUG [U=1065066,P=15499]prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=1065066,P=15499]prepareAutofs() Found "/mnt/NUMEROUS_NFS_MOUNTS" as autofs mount point
DEBUG [U=1065066,P=15499]prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=1065066,P=15499]prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=1065066,P=15499]prepareAutofs() Could not keep file descriptor for home directory /home/REGISTRY/MY_USER: no mount point
DEBUG [U=1065066,P=15499]prepareAutofs() Could not keep file descriptor for current working directory /home/REGISTRY/MY_USER: no mount point
DEBUG [U=1065066,P=15499]Init() Image format detection
DEBUG [U=1065066,P=15499]Init() Check for sandbox image format
DEBUG [U=1065066,P=15499]Init() sandbox image format detected
DEBUG [U=1065066,P=15499]setSessionLayer() Using underlay layer: user namespace requested
VERBOSE [U=1065066,P=15042] wait_child() stage 1 exited with status 0
DEBUG [U=1065066,P=15042] init() Applying stage 1 working directory
DEBUG [U=1065066,P=15042] cleanup_fd() Close file descriptor 4
DEBUG [U=1065066,P=15042] cleanup_fd() Close file descriptor 5
DEBUG [U=1065066,P=15042] cleanup_fd() Close file descriptor 6
DEBUG [U=1065066,P=15042] cleanup_fd() Close file descriptor 7
DEBUG [U=1065066,P=15042] init() Set child signal mask
DEBUG [U=1065066,P=15042] init() Create socketpair for master communication channel
DEBUG [U=1065066,P=15042] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1065066,P=15042] user_namespace_init() Create user namespace
VERBOSE [U=1065066,P=15042] create_namespace() Create user namespace
VERBOSE [U=65534,P=15042] init() Spawn master process
DEBUG [U=65534,P=15042] setup_userns_mappings() Write deny to setgroups file
DEBUG [U=65534,P=15042] setup_userns_mappings() Write to GID map
DEBUG [U=65534,P=15042] setup_userns_mappings() Write to UID map
DEBUG [U=1065066,P=15504] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1065066,P=15504] create_namespace() Create mount namespace
VERBOSE [U=1065066,P=15042] enter_namespace() Entering in mount namespace
DEBUG [U=1065066,P=15042] enter_namespace() Opening namespace file ns/mnt
VERBOSE [U=1065066,P=15504] create_namespace() Create mount namespace
DEBUG [U=1065066,P=15042]startup() singularity runtime engine selected
VERBOSE [U=1065066,P=15042]startup() Execute master process
VERBOSE [U=1065066,P=15507] init() Spawn RPC server
DEBUG [U=1065066,P=15507]startup() singularity runtime engine selected
VERBOSE [U=1065066,P=15507]startup() Serve RPC requests
DEBUG [U=1065066,P=15042]setupSessionLayout() Using Layer system: underlay
DEBUG [U=1065066,P=15042]setupUnderlayLayout() Creating underlay SESSIONDIR layout
DEBUG [U=1065066,P=15042]addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=1065066,P=15042]addRootfsMount() Image type is 4098
DEBUG [U=1065066,P=15042]addRootfsMount() Mounting directory rootfs: /scratch/MY_USER/singularity/tmp/rootfs-503788436/root
DEBUG [U=1065066,P=15042]addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=1065066,P=15042]addKernelMount() Adding proc to mount list
VERBOSE [U=1065066,P=15042]addKernelMount() Default mount: /proc:/proc
DEBUG [U=1065066,P=15042]addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=1065066,P=15042]addKernelMount() Adding sysfs to mount list
VERBOSE [U=1065066,P=15042]addKernelMount() Default mount: /sys:/sys
DEBUG [U=1065066,P=15042]addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=1065066,P=15042]addDevMount() Adding dev to mount list
VERBOSE [U=1065066,P=15042]addDevMount() Default mount: /dev:/dev
DEBUG [U=1065066,P=15042]addHostMount() Not mounting host file systems per configuration
VERBOSE [U=1065066,P=15042]addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
...
...
VERBOSE [U=1065066,P=15042]addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE [U=1065066,P=15042]CleanupContainer() Removing image tempDir /scratch/MY_USER/singularity/tmp/rootfs-503788436
INFO [U=1065066,P=15042]CleanupContainer() Cleaning up image...
FATAL [U=1065066,P=15042]Master() container creation failed: failed to add as session directory: path . is not an absolute path
What OS/distro are you running
$ cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.8 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.8"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.8"
How did you install Singularity
Installed via git, then provided to cluster nodes via a lmod:
$ git clone --recursive https://github.com/hpcng/singularity.git
$ cd singularity
$ module load Go/1.16.6
$ ./mconfig -s -S --prefix=/resource/apps/singularity/installation/ --without-suid
$ make -C builddir
$ make -C builddir install
The module file sets cache and tmp because the head node doesn't have a ton of /tmp space:
help([==[
Description
===========
Singularity is a portable application stack packaging and runtime utility.
More information
================
- Homepage: http://singularity.lbl.gov
]==])
whatis([==[Description: Singularity is a portable application stack packaging and runtime utility.]==])
whatis([==[Homepage: http://singularity.lbl.gov]==])
whatis([==[URL: http://singularity.lbl.gov]==])
local root = "/resource/apps/singularity/installation"
local builder = pathJoin("/scratch",os.getenv('USER'),"singularity")
local tmpFolder = pathJoin(builder,"tmp")
conflict("Singularity")
prepend_path("CMAKE_PREFIX_PATH", root)
--prepend_path("CPATH", pathJoin(root, "include"))
--prepend_path("LIBRARY_PATH", pathJoin(root, "lib"))
--prepend_path("MANPATH", pathJoin(root, "share/man"))
prepend_path("PATH", pathJoin(root, "bin"))
--prepend_path("XDG_DATA_DIRS", pathJoin(root, "share"))
setenv("EBROOTSINGULARITY", root)
setenv("EBVERSIONSINGULARITY", "3.8.3")
--setenv("EBDEVELSINGULARITY", pathJoin(root, "easybuild/Singularity-2.4.2-GCC-5.4.0-2.26-easybuild-devel"))
-- Built with EasyBuild version 4.4.2
-- Cuz /tmp doesn't normally need do be created?
if mode() == 'load' then
local attrBuild= lfs.attributes(builder)
if attrBuild == nill then
lfs.mkdir(builder)
end
local attrTmp = lfs.attributes(tmpFolder)
if attrTmp == nill then
lfs.mkdir(tmpFolder)
end
end
setenv("SINGULARITY_CACHEDIR", pathJoin(builder,"cache"))
setenv("SINGULARITY_TMPDIR", tmpFolder)
You don't change branches after doing 'git clone', so this is actually the current master branch, correct? Do you have different behavior if you next do 'git checkout v3.8.3'.
I cannot follow your instructions because I do not have the 'module' command. Tell me how to use that as well, or better yet, see if you can reproduce it without depending on a module command. You show a module file but I don't know what to do with it.
Sorry. I assumed the current version available on git would be at least 3.8.3 not 3.8.2...
The same issue is occurring in 3.8.3
Source:
$ wget https://github.com/hpcng/singularity/releases/download/v3.8.3/singularity-3.8.3.tar.gz
$ tar -xvf singularity-3.8.3.tar.gz
$ cd singularity-3.8.3
Compiler:
Compiler
$ go env
go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/USER/.cache/go-build"
GOENV="/home/USER/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/USER/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/USER/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/resource/easybuild/software/Go/1.16.6"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/resource/easybuild/software/Go/1.16.6/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.6"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3883183802=/tmp/go-build -gno-record-gcc-switches"
Build Process
$ ./mconfig -s -S --prefix=${PWD}/installation/ --without-suid
Configuring for project `singularity' with languages: C, Golang
=> running pre-basechecks project specific checks ...
=> running base system checks ...
checking: host C compiler... cc
checking: host C++ compiler... c++
checking: host Go compiler (at least version 1.13)... /resource/easybuild/software/Go/1.16.6/bin/go
checking: host C compiler option -Wall... yes
checking: host C compiler option -Werror... yes
checking: host C compiler option -Wfatal-errors... yes
checking: host C compiler option -Wno-unknown-warning-option... yes
checking: host C compiler option -Wstrict-prototypes... yes
checking: host C compiler option -Wpointer-arith... yes
checking: host C compiler option -Wbad-function-cast... yes
checking: host C compiler option -Woverlength-strings... yes
checking: host C compiler option -Wframe-larger-than=2047... yes
checking: host C compiler option -Wno-sign-compare... yes
checking: host C compiler option -Wclobbered... yes
checking: host C compiler option -Wempty-body... yes
checking: host C compiler option -Wmissing-parameter-type... yes
checking: host C compiler option -Wtype-limits... yes
checking: host C compiler option -Wunused-parameter... yes
checking: host C compiler option -Wunused-but-set-parameter... yes
checking: host C compiler option -Wno-discarded-qualifiers... yes
checking: host C compiler option -Wno-incompatible-pointer-types... yes
checking: host C compiler option -pipe... yes
checking: host C compiler option -fmessage-length=0... yes
checking: host C compiler option -fPIC... yes
checking: host `ar' path... ar
checking: host `ld' path... ld
checking: host `ranlib' path... ranlib
checking: host `objcopy' path... objcopy
checking: target C compiler... cc
checking: target C++ compiler... c++
checking: target `ar' path... ar
checking: target `ld' path... ld
checking: target `ranlib' path... ranlib
checking: target `objcopy' path... objcopy
checking: host compiles static binaries... no
checking: target compiles static binaries... no
checking: host os type... unix
checking: host architecture... x86_64
checking: target architecture... x86_64
checking: host architecture word size... 64
checking: target architecture word size... 64
checking: project version... 3.8.3
checking: project short version... 3.8.3
=> running post-basechecks project specific checks ...
checking: namespace: CLONE_NEWPID... yes
checking: namespace: CLONE_FS... yes
checking: namespace: CLONE_NEWNS... yes
checking: namespace: CLONE_NEWUSER... yes
checking: namespace: CLONE_NEWIPC... yes
checking: namespace: CLONE_NEWNET... yes
checking: namespace: CLONE_NEWUTS... yes
checking: namespace: CLONE_NEWCGROUP... no
checking: feature: NO_NEW_PRIVS... yes
checking: feature: MS_SLAVE... yes
checking: feature: MS_REC... yes
checking: feature: MS_PRIVATE... yes
checking: user capabilities... yes
checking: header linux/securebits.h... yes
checking: header linux/capability.h... yes
checking: libseccomp+headers... yes
checking: cryptsetup... yes
=> generating fragments ...
=> building Makefile ...
=> generating singularity.spec ...
=> project singularity setup with :
- host arch: x86_64
- host wordsize: 64-bit
- host C compiler: cc
- host Go compiler: /resource/easybuild/software/Go/1.16.6/bin/go
- host system: unix
---
- target arch: x86_64
- target wordsize: 64-bit
- target C compiler: cc
---
- config profile: release
---
- SUID install: no
- Network plugins: yes
---
- verbose: no
---
- cryptsetup: /sbin/cryptsetup
---
- version: 3.8.3
=> /resource/apps/singularity-3.8.3/builddir/Makefile ready, try:
$ cd /resource/apps/singularity-3.8.3/builddir
$ make
$ make -C builddir
make: Entering directory `/resource/apps/singularity-3.8.3/builddir'
GEN GO DEP /resource/apps/singularity-3.8.3/builddir/starter.d
GEN GO DEP /resource/apps/singularity-3.8.3/builddir/singularity.d
go: downloading github.com/containerd/containerd v1.5.5
go: downloading github.com/opencontainers/runc v1.0.1
make: Leaving directory `/resource/apps/singularity-3.8.3/builddir'
make: Entering directory `/resource/apps/singularity-3.8.3/builddir'
GEN /resource/apps/singularity-3.8.3/scripts/go-generate
GO singularity
[+] GO_TAGS "containers_image_openpgp sylog oci_engine singularity_engine fakeroot_engine apparmor selinux seccomp"
GEN etc/bash_completion.d/singularity
GEN singularity.conf from /resource/apps/singularity-3.8.3/installation//etc/singularity/singularity.conf
CNI PLUGIN dhcp
CNI PLUGIN host-local
CNI PLUGIN static
CNI PLUGIN bridge
CNI PLUGIN host-device
CNI PLUGIN ipvlan
CNI PLUGIN loopback
CNI PLUGIN macvlan
CNI PLUGIN ptp
CNI PLUGIN vlan
CNI PLUGIN bandwidth
CNI PLUGIN firewall
CNI PLUGIN flannel
CNI PLUGIN portmap
CNI PLUGIN tuning
GO clean -cache
GO cmd/starter/c/starter
GEN /resource/apps/singularity-3.8.3/scripts/go-test
make: Leaving directory `/resource/apps/singularity-3.8.3/builddir'
$ make -C builddir install
make: Entering directory `/resource/apps/singularity-3.8.3/builddir'
INSTALL /resource/apps/singularity-3.8.3/installation//bin/singularity
INSTALL /resource/apps/singularity-3.8.3/installation//etc/bash_completion.d/singularity
INSTALL /resource/apps/singularity-3.8.3/installation//etc/singularity/singularity.conf
INSTALL /resource/apps/singularity-3.8.3/installation//etc/singularity/remote.yaml
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/dhcp
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/host-local
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/static
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/bridge
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/host-device
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/ipvlan
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/loopback
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/macvlan
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/ptp
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/vlan
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/bandwidth
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/firewall
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/flannel
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/portmap
INSTALL CNI PLUGIN /resource/apps/singularity-3.8.3/installation//libexec/singularity/cni/tuning
INSTALL CNI CONFIGURATION FILES
INSTALL /resource/apps/singularity-3.8.3/installation//libexec/singularity/bin/starter
INSTALL /resource/apps/singularity-3.8.3/installation//var/singularity/mnt/session
INSTALL /resource/apps/singularity-3.8.3/installation//bin/run-singularity
INSTALL /resource/apps/singularity-3.8.3/installation//etc/singularity/capability.json
INSTALL /resource/apps/singularity-3.8.3/installation//etc/singularity/ecl.toml
INSTALL /resource/apps/singularity-3.8.3/installation//etc/singularity/seccomp-profiles/default.json
INSTALL /resource/apps/singularity-3.8.3/installation//etc/singularity/nvliblist.conf
INSTALL /resource/apps/singularity-3.8.3/installation//etc/singularity/rocmliblist.conf
INSTALL /resource/apps/singularity-3.8.3/installation//etc/singularity/cgroups/cgroups.toml
INSTALL /resource/apps/singularity-3.8.3/installation//etc/singularity/global-pgp-public
DONE
make: Leaving directory `/resource/apps/singularity-3.8.3/builddir'
Version
$ PATH=/resource/apps/singularity-3.8.3/installation//bin/:$PATH
$ singularity --version
singularity version 3.8.3
Environment for local user
$ pwd
/home/USER
$ env | grep singularity
OLDPWD=/resource/apps/singularity-3.8.3
__LMOD_REF_COUNT_LOADEDMODULES=Singularity/3.8.3:1
__LMOD_REF_COUNT_CMAKE_PREFIX_PATH=/resource/apps/singularity-3.8.3/installation:1
EBVERSIONSINGULARITY=3.8.3
__LMOD_REF_COUNT__LMFILES_=/resource/modules_custom/Singularity/3.8.3.lua:1
SINGULARITY_TMPDIR=/scratch/samh/singularity/tmp
PATH=/resource/apps/singularity-3.8.3/installation/bin:/home/USER/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/USER/.local/bin
_LMFILES_=/resource/modules_custom/Singularity/3.8.3.lua
LOADEDMODULES=Singularity/3.8.3
SINGULARITY_CACHEDIR=/scratch/samh/singularity/cache
__LMOD_REF_COUNT_PATH=/resource/apps/singularity-3.8.3/installation/bin:1;/home/samh/bin:1;/usr/lib64/qt-3.3/bin:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/home/samh/.local/bin:1
EBROOTSINGULARITY=/resource/apps/singularity-3.8.3/installation
CMAKE_PREFIX_PATH=/resource/apps/singularity-3.8.3/installation
Run for local user
Run
$ singularity --debug run shub://GodloveD/lolcow
DEBUG [U=1000,P=76267] persistentPreRun() Singularity version: 3.8.3
DEBUG [U=1000,P=76267] persistentPreRun() Parsing configuration file /resource/apps/singularity-3.8.3/installation/etc/singularity/singularity.conf
DEBUG [U=1000,P=76267] handleConfDir() /home/USER/.singularity already exists. Not creating.
DEBUG [U=1000,P=76267] setValue() Updated flag 'tmpdir' value to: /scratch/USER/singularity/tmp
DEBUG [U=1000,P=76267] GetManifest() shub request: https://singularity-hub.org/api/container/GodloveD/lolcow:latest
DEBUG [U=1000,P=76267] GetManifest() 200 OK response received, beginning manifest download
DEBUG [U=1000,P=76267] GetManifest() manifest image name: GodloveD/lolcow
INFO [U=1000,P=76267] pull() Use cached image
DEBUG [U=1000,P=76267] execStarter() Saving umask 0002 for propagation into container
VERBOSE [U=1000,P=76267] execStarter() Unprivileged installation: using user namespace
DEBUG [U=1000,P=76267] execStarter() Checking for encrypted system partition
DEBUG [U=1000,P=76267] Init() Image format detection
DEBUG [U=1000,P=76267] Init() Check for sandbox image format
DEBUG [U=1000,P=76267] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=76267] Init() Check for sif image format
DEBUG [U=1000,P=76267] Init() sif format initializer returned: SIF magic not found
DEBUG [U=1000,P=76267] Init() Check for squashfs image format
DEBUG [U=1000,P=76267] Init() squashfs image format detected
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding MANPATH environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding XDG_SESSION_ID environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding HOSTNAME environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding __LMOD_REF_COUNT_MODULEPATH environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding EB_ARGS environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LMOD_ROOT environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding HISTSIZE environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding MODULEPATH_ROOT environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding SSH_CLIENT environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LMOD_PKG environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding QTDIR environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding OLDPWD environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding EASYBUILD_PREFIX environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LMOD_VERSION environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding QTINC environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding SSH_TTY environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding __LMOD_REF_COUNT_LOADEDMODULES environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding __LMOD_REF_COUNT_CMAKE_PREFIX_PATH environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding EBVERSIONSINGULARITY environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding QT_GRAPHICSSYSTEM_CHECKED environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding http_proxy environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding HISTFILESIZE environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding USER environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LMOD_sys environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LS_COLORS environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding __LMOD_REF_COUNT__LMFILES_ environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding FTP_PROXY environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding ftp_proxy environment variable
VERBOSE [U=1000,P=76267] SetContainerEnv() Not forwarding SINGULARITY_TMPDIR environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding ALL_PROXY environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding _ModuleTable001_ environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding MAIL environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding _LMFILES_ environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding EDITOR environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding MODULEPATH environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LOADEDMODULES environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding _ModuleTable_Sz_ environment variable
VERBOSE [U=1000,P=76267] SetContainerEnv() Not forwarding SINGULARITY_CACHEDIR environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LMOD_CMD environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding SQUEUE_FORMAT environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding HTTPS_PROXY environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding https_proxy environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding HISTCONTROL environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding __LMOD_REF_COUNT_PATH environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding _ModuleTable002_ environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding no_proxy environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding EB_MINIMAL environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding HTTP_PROXY environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding BASH_ENV environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding DSMC_ALL environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding QTLIB environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding SSH_CONNECTION environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding MODULESHOME environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LMOD_SETTARG_FULL_SUPPORT environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LESSOPEN environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding PROMPT_COMMAND environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding EBROOTSINGULARITY environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding CMAKE_PREFIX_PATH environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding DISPLAY environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding LMOD_DIR environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding DSMC_NODES environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding MOD_PATH environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding BASH_FUNC_module() environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding BASH_FUNC_ml() environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding _ environment variable
DEBUG [U=1000,P=76267] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=1000,P=76267] SetContainerEnv() Setting HOME=/home/USER
VERBOSE [U=1000,P=76267] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VERBOSE [U=1000,P=76267] execStarter() User namespace requested, convert image /scratch/USER/singularity/cache/cache/shub/a59d8de3121579fe9c95ab8af0297c2e3aefd827 to sandbox
INFO [U=1000,P=76267] execStarter() Converting SIF file to temporary sandbox...
DEBUG [U=1000,P=76267] Init() Image format detection
DEBUG [U=1000,P=76267] Init() Check for sandbox image format
DEBUG [U=1000,P=76267] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=76267] Init() Check for sif image format
DEBUG [U=1000,P=76267] Init() sif format initializer returned: SIF magic not found
DEBUG [U=1000,P=76267] Init() Check for squashfs image format
DEBUG [U=1000,P=76267] Init() squashfs image format detected
DEBUG [U=1000,P=76267] extract() Excluding /dev directory during root filesystem extraction (non root user)
DEBUG [U=1000,P=76267] extract() Trying unsquashfs options: [-user-xattrs -r]
DEBUG [U=1000,P=76267] unsquashfsSandboxCmd() Calling wrapped unsquashfs: singularity [-q exec --no-home --no-nv --no-rocm -C --no-init --writable -B /scratch/USER/singularity/tmp/rootfs-186733428:/image -B /sbin/unsquashfs:/sbin/unsquashfs:ro -B /lib64/libpthread.so.0:/lib64/libpthread.so.0:ro -B /lib64/libm.so.6:/lib64/libm.so.6:ro -B /lib64/libz.so.1:/lib64/libz.so.1:ro -B /lib64/liblzma.so.5:/lib64/liblzma.so.5:ro -B /lib64/liblzo2.so.2:/lib64/liblzo2.so.2:ro -B /lib64/libgcc_s.so.1:/lib64/libgcc_s.so.1:ro -B /lib64/libc.so.6:/lib64/libc.so.6:ro -B /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2:ro /scratch/USER/singularity/tmp/rootfs-186733428/tmp-rootfs-225925485 /sbin/unsquashfs -user-xattrs -r -d /image/root /image/archive-027936835 ^(.{0}[^d]|.{1}[^e]|.{2}[^v]|.{3}[^\x2f]).*$]
DEBUG [U=1000,P=76267] init() Use starter binary /resource/apps/singularity-3.8.3/installation/libexec/singularity/bin/starter
VERBOSE [U=1000,P=76267] print() Set messagelevel to: 5
VERBOSE [U=1000,P=76267] init() Starter initialization
VERBOSE [U=1000,P=76267] is_suid() Check if we are running as setuid
DEBUG [U=1000,P=76267] read_engine_config() Read engine configuration
DEBUG [U=1000,P=76267] init() Wait completion of stage1
DEBUG [U=1000,P=76372] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=76372] init() Spawn stage 1
DEBUG [U=1000,P=76372] startup() singularity runtime engine selected
VERBOSE [U=1000,P=76372] startup() Execute stage 1
DEBUG [U=1000,P=76372] StageOne() Entering stage 1
DEBUG [U=1000,P=76372] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=1000,P=76372] prepareAutofs() Found "/mnt/A_BUNCH_OF_MOUNTS" as autofs mount point
DEBUG [U=1000,P=76372] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=1000,P=76372] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=1000,P=76372] prepareAutofs() Could not keep file descriptor for home directory /home/USER: no mount point
DEBUG [U=1000,P=76372] prepareAutofs() Could not keep file descriptor for current working directory /home/USER: no mount point
DEBUG [U=1000,P=76372] Init() Image format detection
DEBUG [U=1000,P=76372] Init() Check for sandbox image format
DEBUG [U=1000,P=76372] Init() sandbox image format detected
DEBUG [U=1000,P=76372] setSessionLayer() Using underlay layer: user namespace requested
VERBOSE [U=1000,P=76267] wait_child() stage 1 exited with status 0
DEBUG [U=1000,P=76267] init() Applying stage 1 working directory
DEBUG [U=1000,P=76267] cleanup_fd() Close file descriptor 4
DEBUG [U=1000,P=76267] cleanup_fd() Close file descriptor 5
DEBUG [U=1000,P=76267] cleanup_fd() Close file descriptor 6
DEBUG [U=1000,P=76267] init() Set child signal mask
DEBUG [U=1000,P=76267] init() Create socketpair for master communication channel
DEBUG [U=1000,P=76267] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1000,P=76267] user_namespace_init() Create user namespace
VERBOSE [U=1000,P=76267] create_namespace() Create user namespace
VERBOSE [U=65534,P=76267] init() Spawn master process
DEBUG [U=65534,P=76267] setup_userns_mappings() Write deny to setgroups file
DEBUG [U=65534,P=76267] setup_userns_mappings() Write to GID map
DEBUG [U=65534,P=76267] setup_userns_mappings() Write to UID map
DEBUG [U=1000,P=76378] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=76378] create_namespace() Create mount namespace
VERBOSE [U=1000,P=76267] enter_namespace() Entering in mount namespace
DEBUG [U=1000,P=76267] enter_namespace() Opening namespace file ns/mnt
VERBOSE [U=1000,P=76378] create_namespace() Create mount namespace
VERBOSE [U=1000,P=76379] init() Spawn RPC server
DEBUG [U=1000,P=76267] startup() singularity runtime engine selected
VERBOSE [U=1000,P=76267] startup() Execute master process
DEBUG [U=1000,P=76379] startup() singularity runtime engine selected
VERBOSE [U=1000,P=76379] startup() Serve RPC requests
DEBUG [U=1000,P=76267] setupSessionLayout() Using Layer system: underlay
DEBUG [U=1000,P=76267] setupUnderlayLayout() Creating underlay SESSIONDIR layout
DEBUG [U=1000,P=76267] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=1000,P=76267] addRootfsMount() Image type is 4098
DEBUG [U=1000,P=76267] addRootfsMount() Mounting directory rootfs: /scratch/USER/singularity/tmp/rootfs-186733428/root
DEBUG [U=1000,P=76267] addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=1000,P=76267] addKernelMount() Adding proc to mount list
VERBOSE [U=1000,P=76267] addKernelMount() Default mount: /proc:/proc
DEBUG [U=1000,P=76267] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=1000,P=76267] addKernelMount() Adding sysfs to mount list
VERBOSE [U=1000,P=76267] addKernelMount() Default mount: /sys:/sys
DEBUG [U=1000,P=76267] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=1000,P=76267] addDevMount() Adding dev to mount list
VERBOSE [U=1000,P=76267] addDevMount() Default mount: /dev:/dev
DEBUG [U=1000,P=76267] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=1000,P=76267] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1000,P=76267] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=1000,P=76267] addHomeStagingDir() Staging home directory (/home/USER) at /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/home/USER
DEBUG [U=1000,P=76267] addHomeMount() Adding home directory mount [/resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/home/USER:/home/USER] to list using layer: underlay
DEBUG [U=1000,P=76267] addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=1000,P=76267] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=1000,P=76267] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=1000,P=76267] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=1000,P=76267] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=76267] addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=76267] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=1000,P=76267] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=1000,P=76267] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=1000,P=76267] create() Mount all
DEBUG [U=1000,P=76267] mountGeneric() Mounting tmpfs to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session
DEBUG [U=1000,P=76267] mountGeneric() Mounting /scratch/USER/singularity/tmp/rootfs-186733428/root to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs
DEBUG [U=1000,P=76267] mountGeneric() Remounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs
DEBUG [U=1000,P=76267] mountGeneric() Remounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/bin to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/bin
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/games to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/games
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/include to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/include
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/lib to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/lib
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/local to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/local
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/sbin to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/sbin
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/src to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/src
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/adduser to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/adduser
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/apport to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/apport
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/base-files to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/base-files
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/base-passwd to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/base-passwd
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/bash-completion to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/bash-completion
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/bug to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/bug
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/ca-certificates to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/ca-certificates
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/common-licenses to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/common-licenses
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/cowsay to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/cowsay
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/dbus-1 to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/dbus-1
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/debconf to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/debconf
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/debianutils to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/debianutils
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/dict to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/dict
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/doc to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/doc
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/doc-base to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/doc-base
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/dpkg to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/dpkg
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/fonts to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/fonts
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/games to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/games
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/gcc-5 to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/gcc-5
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/gdb to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/gdb
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/gnupg to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/gnupg
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/info to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/info
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/initramfs-tools to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/initramfs-tools
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/insserv to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/insserv
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/javascript to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/javascript
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/keyrings to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/keyrings
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/libc-bin to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/libc-bin
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/lintian to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/lintian
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/locale to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/locale
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/man to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/man
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/menu to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/menu
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/misc to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/misc
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/pam to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/pam
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/pam-configs to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/pam-configs
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/perl to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/perl
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/perl5 to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/perl5
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/pixmaps to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/pixmaps
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/pkgconfig to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/pkgconfig
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/polkit-1 to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/polkit-1
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/readline to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/readline
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/rubygems-integration to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/rubygems-integration
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/systemd to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/systemd
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/systemtap to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/systemtap
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/sysv-rc to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/sysv-rc
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/tabset to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/tabset
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/terminfo to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/terminfo
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/usr/share/zsh to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/usr/share/zsh
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/.coredone to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/.coredone
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/.singularity.d to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/.singularity.d
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/bin to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/bin
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/boot to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/boot
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/dev to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/dev
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/etc to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/etc
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/lib to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/lib
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/lib64 to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/lib64
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/media to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/media
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/mnt to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/mnt
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/opt to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/opt
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/proc to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/proc
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/root to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/root
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/run to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/run
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/sbin to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/sbin
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/srv to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/srv
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/sys to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/sys
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/tmp to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/tmp
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/var to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay/var
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/underlay to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final
DEBUG [U=1000,P=76267] mountGeneric() Remounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final
DEBUG [U=1000,P=76267] setPropagationMount() Set RPC mount propagation flag to SLAVE
VERBOSE [U=1000,P=76267] Passwd() Checking for template passwd file: /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=1000,P=76267] Passwd() Creating passwd content
VERBOSE [U=1000,P=76267] Passwd() Creating template passwd file and appending user data: /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/etc/passwd
DEBUG [U=1000,P=76267] addIdentityMount() Adding /etc/passwd to mount list
VERBOSE [U=1000,P=76267] addIdentityMount() Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=1000,P=76267] Group() Checking for template group file: /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=1000,P=76267] Group() Creating group content
DEBUG [U=1000,P=76267] addIdentityMount() Adding /etc/group to mount list
VERBOSE [U=1000,P=76267] addIdentityMount() Default mount: /etc/group:/etc/group
DEBUG [U=1000,P=76267] mountGeneric() Mounting /dev to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/dev
DEBUG [U=1000,P=76267] mountGeneric() Mounting /etc/localtime to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC
DEBUG [U=1000,P=76267] mountGeneric() Remounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC
DEBUG [U=1000,P=76267] mountGeneric() Mounting /etc/hosts to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=1000,P=76267] mountGeneric() Remounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=1000,P=76267] mountGeneric() Mounting /proc to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/proc
DEBUG [U=1000,P=76267] mountGeneric() Mounting /sys to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/sys
DEBUG [U=1000,P=76267] mountGeneric() Mounting /home/USER to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/home/USER
DEBUG [U=1000,P=76267] mountGeneric() Remounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/home/USER
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/home/USER to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/home/USER
DEBUG [U=1000,P=76267] mountGeneric() Mounting /tmp to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/tmp
DEBUG [U=1000,P=76267] mountGeneric() Remounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/tmp
DEBUG [U=1000,P=76267] mountGeneric() Mounting /var/tmp to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/var/tmp
DEBUG [U=1000,P=76267] mountGeneric() Remounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/var/tmp
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/etc/resolv.conf to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/etc/resolv.conf
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/etc/passwd to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/etc/passwd
DEBUG [U=1000,P=76267] mountGeneric() Mounting /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/etc/group to /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final/etc/group
DEBUG [U=1000,P=76267] addCwdMount() Using /home/USER as current working directory
VERBOSE [U=1000,P=76267] addCwdMount() /home/USER found within container
DEBUG [U=1000,P=76267] create() Chroot into /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final
DEBUG [U=1000,P=76379] Chroot() Hold reference to host / directory
DEBUG [U=1000,P=76379] Chroot() Called pivot_root on /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/final
DEBUG [U=1000,P=76379] Chroot() Change current directory to host / directory
DEBUG [U=1000,P=76379] Chroot() Apply slave mount propagation for host / directory
DEBUG [U=1000,P=76379] Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=1000,P=76379] Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=1000,P=76267] create() Chdir into / to avoid errors
VERBOSE [U=1000,P=76378] wait_child() rpc server exited with status 0
DEBUG [U=1000,P=76378] init() Set container privileges
DEBUG [U=1000,P=76378] apply_privileges() Effective capabilities: 0x0000000000000000
DEBUG [U=1000,P=76378] apply_privileges() Permitted capabilities: 0x0000000000000000
DEBUG [U=1000,P=76378] apply_privileges() Bounding capabilities: 0x0000000000000000
DEBUG [U=1000,P=76378] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=1000,P=76378] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=1000,P=76378] apply_privileges() Set user ID to 1000
DEBUG [U=1000,P=76378] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=1000,P=76378] startup() singularity runtime engine selected
VERBOSE [U=1000,P=76378] startup() Execute stage 2
DEBUG [U=1000,P=76378] StageTwo() Entering stage 2
DEBUG [U=1000,P=76378] StartProcess() Setting umask in container to 0002
DEBUG [U=1000,P=76378] func1() Not exporting "BASH_FUNC_ml()" to container environment: invalid key
DEBUG [U=1000,P=76378] func1() Not exporting "BASH_FUNC_module()" to container environment: invalid key
DEBUG [U=1000,P=76378] sylogBuiltin() Sourcing /.singularity.d/env/01-base.sh
DEBUG [U=1000,P=76378] sylogBuiltin() Sourcing /.singularity.d/env/10-docker.sh
DEBUG [U=1000,P=76378] sylogBuiltin() Sourcing /.singularity.d/env/90-environment.sh
DEBUG [U=1000,P=76378] sylogBuiltin() Sourcing /.singularity.d/env/95-apps.sh
DEBUG [U=1000,P=76378] sylogBuiltin() Sourcing /.singularity.d/env/99-base.sh
DEBUG [U=1000,P=76378] sylogBuiltin() Running action command run
DEBUG [U=1000,P=76267] PostStartProcess() Post start process
_______________________________________
/ Today is the first day of the rest of \
\ your life. /
---------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
VERBOSE [U=1000,P=76267] CleanupContainer() Removing image tempDir /scratch/USER/singularity/tmp/rootfs-186733428
INFO [U=1000,P=76267] CleanupContainer() Cleaning up image...
DEBUG [U=1000,P=76267] Master() Child exited with exit status 0
Env for DOMAIN user
$ pwd
/home/DOMAIN/USER
$ end | grep singularity
__LMOD_REF_COUNT_LOADEDMODULES=Singularity/3.8.3:1
__LMOD_REF_COUNT_CMAKE_PREFIX_PATH=/resource/apps/singularity-3.8.3/installation:1
EBVERSIONSINGULARITY=3.8.3
__LMOD_REF_COUNT__LMFILES_=/resource/modules_custom/Singularity/3.8.3.lua:1
SINGULARITY_TMPDIR=/scratch/USER/singularity/tmp
PATH=/resource/apps/singularity-3.8.3/installation/bin:/usr/bintercept:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/DOMAIN/USER/.local/bin:/home/DOMAIN/USER/bin
_LMFILES_=/resource/modules_custom/Singularity/3.8.3.lua
LOADEDMODULES=Singularity/3.8.3
SINGULARITY_CACHEDIR=/scratch/USER/singularity/cache
__LMOD_REF_COUNT_PATH=/resource/apps/singularity-3.8.3/installation/bin:1;/usr/bintercept:2;/usr/lib64/qt-3.3/bin:1;/usr/local/bin:3;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/home/DOMAIN/USER/.local/bin:1;/home/DOMAIN/USER/bin:1
EBROOTSINGULARITY=/resource/apps/singularity-3.8.3/installation
CMAKE_PREFIX_PATH=/resource/apps/singularity-3.8.3/installation
Run for DOMAIN user
Run
$ singularity --debug run shub://GodloveD/lolcow
DEBUG [U=1065066,P=5334] persistentPreRun() Singularity version: 3.8.3
DEBUG [U=1065066,P=5334] persistentPreRun() Parsing configuration file /resource/apps/singularity-3.8.3/installation/etc/singularity/singularity.conf
DEBUG [U=1065066,P=5334] handleConfDir() /home/DOMAIN/USER/.singularity already exists. Not creating.
DEBUG [U=1065066,P=5334] setValue() Updated flag 'tmpdir' value to: /scratch/USER/singularity/tmp
DEBUG [U=1065066,P=5334] GetManifest() shub request: https://singularity-hub.org/api/container/GodloveD/lolcow:latest
DEBUG [U=1065066,P=5334] GetManifest() 200 OK response received, beginning manifest download
DEBUG [U=1065066,P=5334] GetManifest() manifest image name: GodloveD/lolcow
INFO [U=1065066,P=5334] pull() Use cached image
DEBUG [U=1065066,P=5334] execStarter() Saving umask 0022 for propagation into container
VERBOSE [U=1065066,P=5334] execStarter() Unprivileged installation: using user namespace
DEBUG [U=1065066,P=5334] execStarter() Checking for encrypted system partition
DEBUG [U=1065066,P=5334] Init() Image format detection
DEBUG [U=1065066,P=5334] Init() Check for sandbox image format
DEBUG [U=1065066,P=5334] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1065066,P=5334] Init() Check for sif image format
DEBUG [U=1065066,P=5334] Init() sif format initializer returned: SIF magic not found
DEBUG [U=1065066,P=5334] Init() Check for squashfs image format
DEBUG [U=1065066,P=5334] Init() squashfs image format detected
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding MANPATH environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding XDG_SESSION_ID environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding HOSTNAME environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding __LMOD_REF_COUNT_MODULEPATH environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LMOD_ROOT environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding HISTSIZE environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding SSH_CLIENT environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding MODULEPATH_ROOT environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LMOD_PKG environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding QTDIR environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding QTINC environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LMOD_VERSION environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding SSH_TTY environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding __LMOD_REF_COUNT_LOADEDMODULES environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding __LMOD_REF_COUNT_CMAKE_PREFIX_PATH environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding POWERLINE_COMMAND environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding EBVERSIONSINGULARITY environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding QT_GRAPHICSSYSTEM_CHECKED environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding http_proxy environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding USER environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding HISTFILESIZE environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LS_COLORS environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LMOD_sys environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding __LMOD_REF_COUNT__LMFILES_ environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding ftp_proxy environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding FTP_PROXY environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding COBBLER_SERVER environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding SCRDIR environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding TMUX environment variable
VERBOSE [U=1065066,P=5334] SetContainerEnv() Not forwarding SINGULARITY_TMPDIR environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding ALL_PROXY environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding _ModuleTable001_ environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding MAIL environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding _LMFILES_ environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding MODULEPATH environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LOADEDMODULES environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding _ModuleTable_Sz_ environment variable
VERBOSE [U=1065066,P=5334] SetContainerEnv() Not forwarding SINGULARITY_CACHEDIR environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding TMUX_PANE environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LMOD_CMD environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding https_proxy environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding HTTPS_PROXY environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding HISTCONTROL environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding __LMOD_REF_COUNT_PATH environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding _ModuleTable002_ environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding no_proxy environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding HTTP_PROXY environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding BASH_ENV environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding QTLIB environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding SSH_CONNECTION environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding MODULESHOME environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LMOD_SETTARG_FULL_SUPPORT environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LESSOPEN environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding PROMPT_COMMAND environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding EBROOTSINGULARITY environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding CMAKE_PREFIX_PATH environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding DISPLAY environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding LMOD_DIR environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding SCRATCH environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding MOD_PATH environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding BASH_FUNC_module() environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding BASH_FUNC_ml() environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding _ environment variable
DEBUG [U=1065066,P=5334] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=1065066,P=5334] SetContainerEnv() Setting HOME=/home/DOMAIN/USER
VERBOSE [U=1065066,P=5334] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VERBOSE [U=1065066,P=5334] execStarter() User namespace requested, convert image /scratch/USER/singularity/cache/cache/shub/a59d8de3121579fe9c95ab8af0297c2e3aefd827 to sandbox
INFO [U=1065066,P=5334] execStarter() Converting SIF file to temporary sandbox...
DEBUG [U=1065066,P=5334] Init() Image format detection
DEBUG [U=1065066,P=5334] Init() Check for sandbox image format
DEBUG [U=1065066,P=5334] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1065066,P=5334] Init() Check for sif image format
DEBUG [U=1065066,P=5334] Init() sif format initializer returned: SIF magic not found
DEBUG [U=1065066,P=5334] Init() Check for squashfs image format
DEBUG [U=1065066,P=5334] Init() squashfs image format detected
DEBUG [U=1065066,P=5334] extract() Excluding /dev directory during root filesystem extraction (non root user)
DEBUG [U=1065066,P=5334] extract() Trying unsquashfs options: [-no-xattrs -r]
DEBUG [U=1065066,P=5334] unsquashfsSandboxCmd() Calling wrapped unsquashfs: singularity [-q exec --no-home --no-nv --no-rocm -C --no-init --writable -B /scratch/USER/singularity/tmp/rootfs-531163379:/image -B /sbin/unsquashfs:/sbin/unsquashfs:ro -B /usr/lib64/libpthread.so.0:/usr/lib64/libpthread.so.0:ro -B /usr/lib64/libm.so.6:/usr/lib64/libm.so.6:ro -B /usr/lib64/libz.so.1:/usr/lib64/libz.so.1:ro -B /usr/lib64/liblzma.so.5:/usr/lib64/liblzma.so.5:ro -B /usr/lib64/liblzo2.so.2:/usr/lib64/liblzo2.so.2:ro -B /usr/lib64/libgcc_s.so.1:/usr/lib64/libgcc_s.so.1:ro -B /usr/lib64/libc.so.6:/usr/lib64/libc.so.6:ro -B /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2:ro /scratch/USER/singularity/tmp/rootfs-531163379/tmp-rootfs-111951448 /sbin/unsquashfs -no-xattrs -r -d /image/root /image/archive-374084022 ^(.{0}[^d]|.{1}[^e]|.{2}[^v]|.{3}[^\x2f]).*$]
DEBUG [U=1065066,P=5334] init() Use starter binary /resource/apps/singularity-3.8.3/installation/libexec/singularity/bin/starter
VERBOSE [U=1065066,P=5334] print() Set messagelevel to: 5
VERBOSE [U=1065066,P=5334] init() Starter initialization
VERBOSE [U=1065066,P=5334] is_suid() Check if we are running as setuid
DEBUG [U=1065066,P=5334] read_engine_config() Read engine configuration
DEBUG [U=1065066,P=5334] init() Wait completion of stage1
DEBUG [U=1065066,P=5810] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1065066,P=5810] init() Spawn stage 1
DEBUG [U=1065066,P=5810] startup() singularity runtime engine selected
VERBOSE [U=1065066,P=5810] startup() Execute stage 1
DEBUG [U=1065066,P=5810] StageOne() Entering stage 1
DEBUG [U=1065066,P=5810] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=1065066,P=5810] prepareAutofs() Found "/mnt/A_BUNCH_OF_MOUNTS" as autofs mount point
DEBUG [U=1065066,P=5810] prepareAutofs() Found "/mnt/A_BUNCH_OF_MOUNTS" as autofs mount point
DEBUG [U=1065066,P=5810] prepareAutofs() Found "/mnt/A_BUNCH_OF_MOUNTS" as autofs mount point
DEBUG [U=1065066,P=5810] prepareAutofs() Found "/mnt/A_BUNCH_OF_MOUNTS" as autofs mount point
DEBUG [U=1065066,P=5810] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=1065066,P=5810] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=1065066,P=5810] prepareAutofs() Could not keep file descriptor for home directory /home/DOMAIN/USER: no mount point
DEBUG [U=1065066,P=5810] prepareAutofs() Could not keep file descriptor for current working directory /home/DOMAIN/USER: no mount point
DEBUG [U=1065066,P=5810] Init() Image format detection
DEBUG [U=1065066,P=5810] Init() Check for sandbox image format
DEBUG [U=1065066,P=5810] Init() sandbox image format detected
DEBUG [U=1065066,P=5810] setSessionLayer() Using underlay layer: user namespace requested
VERBOSE [U=1065066,P=5334] wait_child() stage 1 exited with status 0
DEBUG [U=1065066,P=5334] init() Applying stage 1 working directory
DEBUG [U=1065066,P=5334] cleanup_fd() Close file descriptor 4
DEBUG [U=1065066,P=5334] cleanup_fd() Close file descriptor 5
DEBUG [U=1065066,P=5334] cleanup_fd() Close file descriptor 6
DEBUG [U=1065066,P=5334] cleanup_fd() Close file descriptor 7
DEBUG [U=1065066,P=5334] init() Set child signal mask
DEBUG [U=1065066,P=5334] init() Create socketpair for master communication channel
DEBUG [U=1065066,P=5334] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1065066,P=5334] user_namespace_init() Create user namespace
VERBOSE [U=1065066,P=5334] create_namespace() Create user namespace
VERBOSE [U=65534,P=5334] init() Spawn master process
DEBUG [U=65534,P=5334] setup_userns_mappings() Write deny to setgroups file
DEBUG [U=65534,P=5334] setup_userns_mappings() Write to GID map
DEBUG [U=65534,P=5334] setup_userns_mappings() Write to UID map
DEBUG [U=1065066,P=5815] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1065066,P=5815] create_namespace() Create mount namespace
VERBOSE [U=1065066,P=5334] enter_namespace() Entering in mount namespace
DEBUG [U=1065066,P=5334] enter_namespace() Opening namespace file ns/mnt
VERBOSE [U=1065066,P=5815] create_namespace() Create mount namespace
DEBUG [U=1065066,P=5334] startup() singularity runtime engine selected
VERBOSE [U=1065066,P=5334] startup() Execute master process
VERBOSE [U=1065066,P=5818] init() Spawn RPC server
DEBUG [U=1065066,P=5818] startup() singularity runtime engine selected
VERBOSE [U=1065066,P=5818] startup() Serve RPC requests
DEBUG [U=1065066,P=5334] setupSessionLayout() Using Layer system: underlay
DEBUG [U=1065066,P=5334] setupUnderlayLayout() Creating underlay SESSIONDIR layout
DEBUG [U=1065066,P=5334] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=1065066,P=5334] addRootfsMount() Image type is 4098
DEBUG [U=1065066,P=5334] addRootfsMount() Mounting directory rootfs: /scratch/USER/singularity/tmp/rootfs-531163379/root
DEBUG [U=1065066,P=5334] addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=1065066,P=5334] addKernelMount() Adding proc to mount list
VERBOSE [U=1065066,P=5334] addKernelMount() Default mount: /proc:/proc
DEBUG [U=1065066,P=5334] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=1065066,P=5334] addKernelMount() Adding sysfs to mount list
VERBOSE [U=1065066,P=5334] addKernelMount() Default mount: /sys:/sys
DEBUG [U=1065066,P=5334] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=1065066,P=5334] addDevMount() Adding dev to mount list
VERBOSE [U=1065066,P=5334] addDevMount() Default mount: /dev:/dev
DEBUG [U=1065066,P=5334] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=1065066,P=5334] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1065066,P=5334] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE [U=1065066,P=5334] CleanupContainer() Removing image tempDir /scratch/USER/singularity/tmp/rootfs-531163379
INFO [U=1065066,P=5334] CleanupContainer() Cleaning up image...
FATAL [U=1065066,P=5334] Master() container creation failed: failed to add as session directory: path . is not an absolute path
I noticed that there are different unsquashfs options -user-xattrs vs no-xattrs. The latter happens when Singularity canot set a user extended attribute on the destination directory. So that is one difference between the two directories, although I don't know if it is significant for triggering the problem.
I notice also that the user id that this is running under is different, 1000 vs 1065066.
The first missing message in the broken run is
Staging home directory (/home/USER) at /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/home/USER
The error is getting tripped in the addHomeStaging() function because the given "dest" directory is empty. That comes from the getHomePaths() function. It seems most likely that it is getting an error in the CurrentOriginal() function although it is hard to tell.
I don't think this is something that anybody else is going to be able to reproduce, and there isn't enough debugging info, so I suggest that you put in some more debug statements and rebuild in order to narrow down the source of the problem.
It looks like lookupUnixUid() is throwing No such file or directory... which is not making it all the back to debug output...
Except that this is the 2nd instance of lookupUnixUid() to run. The first one, somewhere after StageOne() obtains the user id, home folder and shell correctly where as the second, repeat call from lookupUnixUid() does not.
With a bunch of debug output added, for a domain user, I get...
...
DEBUG [U=1065066,P=113245]StageOne() Entering stage 1
DEBUG [U=1065066,P=113245]current() syscall.Getuid() 1065066
DEBUG [U=1065066,P=113245]lookupUnixUid() lookupUnixUid uid: 1065066
DEBUG [U=1065066,P=113245]lookupUnixUid() retryWithBuffer err: <nil>
DEBUG [U=1065066,P=113245]lookupUnixUid() buildUser(&pwd): &{USER 1065066 1000513 /home/DOMAIN/USER /bin/bash}
DEBUG [U=1065066,P=113245]Current() current():&{%!d(string=USER) 1065066 1000513 %!d(string=) %!d(string=/home/DOMAIN/USER) %!d(string=/bin/bash)} err:<nil>
DEBUG [U=1065066,P=113245]prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
...
...and for the local user, an equally sane looking...
...
DEBUG [U=1000,P=28012] StageOne() Entering stage 1
DEBUG [U=1000,P=28012] current() syscall.Getuid() 1000
DEBUG [U=1000,P=28012] lookupUnixUid() lookupUnixUid uid: 1000
DEBUG [U=1000,P=28012] lookupUnixUid() retryWithBuffer err: <nil>
DEBUG [U=1000,P=28012] lookupUnixUid() buildUser(&pwd): &{USER 1000 1001 /home/USER /bin/bash}
DEBUG [U=1000,P=28012] Current() current():&{%!d(string=USER) 1000 1001 %!d(string=) %!d(string=/home/USER) %!d(string=/bin/bash)} err:<nil>
DEBUG [U=1000,P=28012] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
...
Then the second time it calls lookupUnixUid() for the domain user I get...
...
VERBOSE [U=1065066,P=113140]addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=1065066,P=113140]CurrentOriginal() namespaces.HostUID() uid:1065066 err:<nil>
DEBUG [U=1065066,P=113140]GetPwUID() uid 1065066
DEBUG [U=1065066,P=113140]lookupUnixUid() lookupUnixUid uid: 1065066
DEBUG [U=1065066,P=113140]lookupUnixUid() retryWithBuffer err: no such file or directory
DEBUG [U=1065066,P=113140]lookupUnixUid() user: lookup userid 1065066: no such file or directory
DEBUG [U=1065066,P=113140]CurrentOriginal() GetPwUID(uint32(uid)):0 err:user: lookup userid 1065066: no such file or directory
DEBUG [U=1065066,P=113140]getHomePaths() user.CurrentOriginal() pw:<nil> e:user: lookup userid 1065066: no such file or directory
VERBOSE [U=1065066,P=113140]CleanupContainer() Removing image tempDir /scratch/USER/singularity/tmp/rootfs-366957831
...
...vs a local user getting...
...
VERBOSE [U=1000,P=27484] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=1000,P=27484] CurrentOriginal() namespaces.HostUID() uid:1000 err:<nil>
DEBUG [U=1000,P=27484] GetPwUID() uid 1000
DEBUG [U=1000,P=27484] lookupUnixUid() lookupUnixUid uid: 1000
DEBUG [U=1000,P=27484] lookupUnixUid() retryWithBuffer err: <nil>
DEBUG [U=1000,P=27484] lookupUnixUid() buildUser(&pwd): &{USER 1000 1001 /home/USER /bin/bash}
DEBUG [U=1000,P=27484] CurrentOriginal() GetPwUID(uint32(uid)):&{%!d(string=USER) 1000 1001 %!d(string=) %!d(string=/home/USER) %!d(string=/bin/bash)} err:<nil>
DEBUG [U=1000,P=27484] getHomePaths() user.CurrentOriginal() pw:&{USER 1000 1001 /home/USER /bin/bash} e:<nil>
DEBUG [U=1000,P=27484] addHomeStagingDir() Staging home directory (/home/USER) at /resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/home/USER
DEBUG [U=1000,P=27484] addHomeMount() Adding home directory mount [/resource/apps/singularity-3.8.3/installation/var/singularity/mnt/session/home/USER:/home/USER] to list using layer: underlay
DEBUG [U=1000,P=27484] addTmpMount() Checking for 'mount tmp' in configuration file
...
So the same function call with same input works the first time for both local and domain users, but for the domain user it fails the second time it's called.
Surely just retrieving the HOME env variable would suffice?
Hi @shawarden , did it also happen with 3.7/3.6 ... ? It would help to determine is there was a regression at some point. At first glance this is really weird, it means that for the second call (which also occurs in a different process) NSS libraries are not loaded/found so Singularity can only deal with local users.
Surely just retrieving the HOME env variable would suffice?
No, HOME is not a source of trust and here we need to honor potential admin restriction which could be set through singularity configuration file like user bind control = no, and using HOME here like suggested would allow to bypass this restriction.
Hi @cclerget , I haven't used singularity before now.
I have just installed versions going back through 3.3.0 and they all fail for domain users with the same message while running fine with a local user.
2.x versions fail on downloading the simg, asking me if I can write to the current directory. This occurs for both local and domain users and I can write there since it's either my home or scratch area. Both of which I use frequently and without issue.
DEBUG GET (stream) https://datasets.datalad.org/shub/GodloveD/lolcow/latest/2021-04-19-a59d8de3-ee4aae1e/ee4aae1ea378ad7c0299b308c703187a.simg
ERROR Error downloading https://datasets.datalad.org/shub/GodloveD/lolcow/latest/2021-04-19-a59d8de3-ee4aae1e/ee4aae1ea378ad7c0299b308c703187a.simg. Do you have permission to write to /home/DOMAIN/USER?
ABORT: Aborting with RETVAL=255
I ran it on the machine where these drives are physically mounted incase there were NFS export issues.
Singularity repo is now retired as the code base is now moved to Apptainer. We are closing all the old issues under the old Singularity repo. For further assistance please open a new issue under the new Apptainer repo. Thanks for your support.
