singularity
singularity copied to clipboard
Published
20 hours ago •
sylabs
sylabs
singularity run --nv doesn't give access to A100 on Ubuntu 20.04 host w/ CUDA 11.4
Version of Singularity
$ singularity --version
3.9.3
Describe the bug
singularity run --nv e4s-gpu-x86_64.sif fails to give access to A100. Before upgrading to 3.9.3 I was able to launch this same image in the same way and get expected access to A100.
Here is the expected output of nvidia-smi on the host:
Mon Jan 24 08:35:04 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100 80G... Off | 00000000:24:00.0 Off | 0 |
| N/A 35C P0 61W / 300W | 0MiB / 80994MiB | 5% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Here is what I see in the container:
$> singularity run --nv e4s-gpu-x86_64.sif
Singularity> nvidia-smi
No devices were found
Singularity> ldconfig -p | grep cuda
libnvrtc.so.11.2 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvrtc.so.11.2
libnvrtc.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvrtc.so
libnvrtc-builtins.so.11.4 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvrtc-builtins.so.11.4
libnvrtc-builtins.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvrtc-builtins.so
libnvjpeg.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvjpeg.so.11
libnvjpeg.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvjpeg.so
libnvblas.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvblas.so.11
libnvblas.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvblas.so
libnvToolsExt.so.1 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvToolsExt.so.1
libnvToolsExt.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnvToolsExt.so
libnpps.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnpps.so.11
libnpps.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnpps.so
libnppitc.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppitc.so.11
libnppitc.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppitc.so
libnppisu.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppisu.so.11
libnppisu.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppisu.so
libnppist.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppist.so.11
libnppist.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppist.so
libnppim.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppim.so.11
libnppim.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppim.so
libnppig.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppig.so.11
libnppig.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppig.so
libnppif.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppif.so.11
libnppif.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppif.so
libnppidei.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppidei.so.11
libnppidei.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppidei.so
libnppicc.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppicc.so.11
libnppicc.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppicc.so
libnppial.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppial.so.11
libnppial.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppial.so
libnppc.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppc.so.11
libnppc.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libnppc.so
libicudata.so.66 (libc6,x86-64) => /lib/x86_64-linux-gnu/libicudata.so.66
libicudata.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libicudata.so
libcusparse.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcusparse.so.11
libcusparse.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcusparse.so
libcusolverMg.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcusolverMg.so.11
libcusolverMg.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcusolverMg.so
libcusolver.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcusolver.so.11
libcusolver.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcusolver.so
libcurand.so.10 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcurand.so.10
libcurand.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcurand.so
libcuinj64.so.11.4 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcuinj64.so.11.4
libcuinj64.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcuinj64.so
libcufftw.so.10 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcufftw.so.10
libcufftw.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcufftw.so
libcufft.so.10 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcufft.so.10
libcufft.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcufft.so
libcudart.so.11.0 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudart.so.11.0
libcudart.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudart.so
libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
libcuda.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so
libcublasLt.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcublasLt.so.11
libcublasLt.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcublasLt.so
libcublas.so.11 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcublas.so.11
libcublas.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcublas.so
libaccinj64.so.11.4 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libaccinj64.so.11.4
libaccinj64.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libaccinj64.so
libOpenCL.so.1 (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libOpenCL.so.1
libOpenCL.so (libc6,x86-64) => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libOpenCL.so
Singularity> ls -lah /dev/nv*
crw-rw-rw- 1 root root 195, 0 Jan 12 15:38 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Jan 12 15:38 /dev/nvidiactl
crw-rw-rw- 1 root root 496, 0 Jan 12 23:58 /dev/nvidia-uvm
crw-rw-rw- 1 root root 496, 1 Jan 12 23:58 /dev/nvidia-uvm-tools
crw------- 1 root root 241, 0 Jan 12 15:34 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Jan 12 15:34 /dev/nvme0n1
brw-rw---- 1 root disk 259, 1 Jan 12 15:34 /dev/nvme0n1p1
brw-rw---- 1 root disk 259, 2 Jan 12 15:34 /dev/nvme0n1p2
brw-rw---- 1 root disk 259, 3 Jan 12 15:34 /dev/nvme0n1p3
brw-rw---- 1 root disk 259, 4 Jan 12 15:34 /dev/nvme0n1p4
brw-rw---- 1 root disk 259, 5 Jan 12 15:34 /dev/nvme0n1p5
crw------- 1 root root 10, 144 Jan 12 15:34 /dev/nvram
/dev/nvidia-caps:
total 0
drwxr-xr-x 2 root root 80 Jan 12 15:38 .
drwxr-xr-x 24 root root 4.8K Jan 20 13:15 ..
cr-------- 1 root root 503, 1 Jan 12 15:38 nvidia-cap1
cr--r--r-- 1 root root 503, 2 Jan 12 15:38 nvidia-cap2
To Reproduce On Ubuntu 20.04, Singularity 3.9.3, A100, CUDA 11.4:
$> wget https://oaciss.uoregon.edu/e4s/images/21.11/e4s-gpu-x86_64.sif
$> singularity run --nv e4s-gpu-x86_64.sif
Singularity> nvidia-smi
No devices were found
Expected behavior
I would expect nvidia-smi inside the container to yield the same output as nvidia-smi outside the container.
OS / Linux Distribution Which Linux distribution are you using?
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Installation Method Installed Singularity 3.9.3 from source according to these instructions https://sylabs.io/guides/3.9/user-guide/quick_start.html#quick-installation-steps
Additional context
This same image can be launched singularity run --nv e4s-gpu-x86_64.sif on our CentOS 8 host, using Singularity 3.9.3, and there I get the expected output of nvidia-smi showing the A100s available.
Thanks for the information. Noting here that you reported in Slack that it also works as expected when using --nvccli functionality - just not --nv alone.... which should narrow it down a bit.
The following information will be critical to tracking this down, as I cannot reproduce when running a GPU image on an Ubuntu 20.04 system with a GTX card.
-
Full debug output from running
singularitywith the-dflag. I.E.singularity run --nv e4s-gpu-x86_64.sif -
On the host, output of
ldconfig -p | grep libnvidia(to see the basic driver related libraries there). -
The exact method used to install the NVIDIA driver and CUDA, i.e. exact
.runpackage name, OS package version etc. -
A SIF file reproducing the issue that is a lot smaller... I'm afraid that a 41.3GB SIF isn't easily to work with in our test environments.
Output of singularity run -d --nv e4s-gpu-x86_64.sif
DEBUG [U=15702,P=1972264]persistentPreRun() Singularity version: 3.9.3
DEBUG [U=15702,P=1972264]persistentPreRun() Parsing configuration file /usr/local/etc/singularity/singularity.conf
DEBUG [U=15702,P=1972264]handleConfDir() /home/users/lpeyrala/.singularity already exists. Not creating.
DEBUG [U=15702,P=1972264]execStarter() Saving umask 0022 for propagation into container
DEBUG [U=15702,P=1972264]execStarter() Checking for encrypted system partition
DEBUG [U=15702,P=1972264]Init() Image format detection
DEBUG [U=15702,P=1972264]Init() Check for sandbox image format
DEBUG [U=15702,P=1972264]Init() sandbox format initializer returned: not a directory image
DEBUG [U=15702,P=1972264]Init() Check for sif image format
DEBUG [U=15702,P=1972264]Init() sif image format detected
DEBUG [U=15702,P=1972264]setNVLegacyConfig() Using legacy binds for nv GPU setup
VERBOSE [U=15702,P=1972264]NvidiaIpcsPath() persistenced socket /var/run/nvidia-persistenced/socket not found
DEBUG [U=15702,P=1972264]findFromConfigOnly() Using "ldconfig" at "/usr/sbin/ldconfig.real" (from singularity.conf)
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding CONDA_EXE environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding _CE_M environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding ENV environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding XDG_SESSION_TYPE environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding CONDA_PREFIX environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding MODULESHOME environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding MANPATH environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding MOTD_SHOWN environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding CONDA_PROMPT_MODIFIER environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding LC_TERMINAL environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding SSH_CONNECTION environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding DOCKER_CLI_EXPERIMENTAL environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding MODULEPATH_modshare environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding XDG_SESSION_CLASS environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding _CE_CONDA environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding USER environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding CONDA_SHLVL environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding LOADEDMODULES environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding LC_TERMINAL_VERSION environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding BASH_ENV environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding XDG_SESSION_ID environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding CONDA_PYTHON_EXE environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding GO111MODULE environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding SSH_CLIENT environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding CONDA_DEFAULT_ENV environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding XDG_DATA_DIRS environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding MODULEPATH environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding DBUS_SESSION_BUS_ADDRESS environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding SSH_TTY environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding MODULES_CMD environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding BASH_FUNC_switchml%% environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding BASH_FUNC_module%% environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding BASH_FUNC__module_raw%% environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding _ environment variable
DEBUG [U=15702,P=1972264]SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=15702,P=1972264]SetContainerEnv() Setting HOME=/home/users/lpeyrala
VERBOSE [U=15702,P=1972264]SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=15702,P=1972264]init() Use starter binary /usr/local/libexec/singularity/bin/starter-suid
DEBUG [U=15702,P=1972264]Exec() Setting GOGC=off for starter
VERBOSE [U=0,P=1972264] print() Set messagelevel to: 5
VERBOSE [U=0,P=1972264] init() Starter initialization
DEBUG [U=0,P=1972264] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=1972264] load_overlay_module() Overlay seems supported by the kernel
VERBOSE [U=0,P=1972264] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=1972264] priv_drop() Drop root privileges
DEBUG [U=15702,P=1972264] read_engine_config() Read engine configuration
DEBUG [U=15702,P=1972264] cleanenv() Clearing env var LOAD_OVERLAY_MODULE=1
DEBUG [U=15702,P=1972264] cleanenv() Keeping env var SINGULARITY_MESSAGELEVEL=95
DEBUG [U=15702,P=1972264] cleanenv() Clearing env var ENGINE_CONFIG1={"engineName":"singularity","containerID":"","engineConfig":{"jsonConfig":{"filesPath":["/usr/bin/nvidia-smi:/usr/bin/nvidia-smi","/usr/bin/nvidia-debugdump:/usr/bin/nvidia-debugdump","/usr/bin/nvidia-persistenced:/usr/bin/nvidia-persistenced","/usr/bin/nvidia-cuda-mps-control:/usr/bin/nvidia-cuda-mps-control","/usr/bin/nvidia-cuda-mps-server:/usr/bin/nvidia-cuda-mps-server"],"librariesPath":["/lib/x86_64-linux-gnu/libcuda.so","/lib/x86_64-linux-gnu/libcuda.so(TRUNCATED...)DEBUG [U=15702,P=1972264] cleanenv() Clearing env var ENGINE_CONFIG_CHUNKS=1
DEBUG [U=15702,P=1972264] cleanenv() Keeping env var GOGC=off
DEBUG [U=15702,P=1972264] init() Wait completion of stage1
VERBOSE [U=15702,P=1972283] priv_drop() Drop root privileges permanently
DEBUG [U=15702,P=1972283] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=15702,P=1972283] init() Spawn stage 1
DEBUG [U=15702,P=1972283]startup() singularity runtime engine selected
VERBOSE [U=15702,P=1972283]startup() Execute stage 1
DEBUG [U=15702,P=1972283]StageOne() Entering stage 1
DEBUG [U=15702,P=1972283]prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=15702,P=1972283]prepareAutofs() Found "/storage" as autofs mount point
DEBUG [U=15702,P=1972283]prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=15702,P=1972283]prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=15702,P=1972283]keepAutofsMount() Open file descriptor for /storage/users/lpeyrala
DEBUG [U=15702,P=1972283]keepAutofsMount() Open file descriptor for /storage/users/lpeyrala
DEBUG [U=15702,P=1972283]Init() Image format detection
DEBUG [U=15702,P=1972283]Init() Check for sandbox image format
DEBUG [U=15702,P=1972283]Init() sandbox format initializer returned: not a directory image
DEBUG [U=15702,P=1972283]Init() Check for sif image format
DEBUG [U=15702,P=1972283]Init() sif image format detected
DEBUG [U=15702,P=1972283]setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=15702,P=1972283]setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=15702,P=1972264] wait_child() stage 1 exited with status 0
DEBUG [U=15702,P=1972264] cleanup_fd() Setting FD_CLOEXEC on starter fd 3
DEBUG [U=15702,P=1972264] cleanup_fd() Close file descriptor 4
DEBUG [U=15702,P=1972264] cleanup_fd() Close file descriptor 5
DEBUG [U=15702,P=1972264] cleanup_fd() Close file descriptor 6
DEBUG [U=15702,P=1972264] cleanup_fd() Close file descriptor 7
DEBUG [U=15702,P=1972264] cleanup_fd() Setting FD_CLOEXEC on starter fd 8
DEBUG [U=15702,P=1972264] cleanup_fd() Close file descriptor 9
DEBUG [U=15702,P=1972264] cleanup_fd() Setting FD_CLOEXEC on starter fd 10
DEBUG [U=15702,P=1972264] init() Set child signal mask
DEBUG [U=15702,P=1972264] init() Create socketpair for master communication channel
DEBUG [U=15702,P=1972264] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=15702,P=1972264] priv_escalate() Get root privileges
VERBOSE [U=0,P=1972264] priv_escalate() Change filesystem uid to 15702
VERBOSE [U=0,P=1972264] init() Spawn master process
DEBUG [U=0,P=1972289] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=1972289] create_namespace() Create mount namespace
VERBOSE [U=0,P=1972264] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=1972264] enter_namespace() Opening namespace file ns/mnt
DEBUG [U=0,P=1972264] set_master_privileges() Set master privileges
VERBOSE [U=0,P=1972289] create_namespace() Create mount namespace
DEBUG [U=0,P=1972264] apply_privileges() Effective capabilities: 0x00000000000000c0
DEBUG [U=0,P=1972264] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=1972264] apply_privileges() Bounding capabilities: 0x0000003fffffffff
DEBUG [U=0,P=1972264] apply_privileges() Inheritable capabilities: 0x0000003fffffffff
DEBUG [U=0,P=1972264] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=1972264] apply_privileges() Set user ID to 15702
DEBUG [U=0,P=1972290] set_rpc_privileges() Set RPC privileges
DEBUG [U=0,P=1972290] apply_privileges() Effective capabilities: 0x0000000000200000
DEBUG [U=0,P=1972290] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=1972290] apply_privileges() Bounding capabilities: 0x0000000008204000
DEBUG [U=0,P=1972290] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=1972290] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=1972290] apply_privileges() Set user ID to 15702
DEBUG [U=15702,P=1972290] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=15702,P=1972290] init() Spawn RPC server
DEBUG [U=15702,P=1972264]startup() singularity runtime engine selected
VERBOSE [U=15702,P=1972264]startup() Execute master process
DEBUG [U=15702,P=1972264]setupSessionLayout() Using Layer system: overlay
DEBUG [U=15702,P=1972264]setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=15702,P=1972264]addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=15702,P=1972264]addRootfsMount() Image type is 4096
DEBUG [U=15702,P=1972264]addRootfsMount() Mounting block [squashfs] image: /home/users/lpeyrala/e4s-gpu-x86_64.sif
DEBUG [U=15702,P=1972264]addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=15702,P=1972264]addKernelMount() Adding proc to mount list
VERBOSE [U=15702,P=1972264]addKernelMount() Default mount: /proc:/proc
DEBUG [U=15702,P=1972264]addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=15702,P=1972264]addKernelMount() Adding sysfs to mount list
VERBOSE [U=15702,P=1972264]addKernelMount() Default mount: /sys:/sys
DEBUG [U=15702,P=1972264]addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=15702,P=1972264]addDevMount() Adding dev to mount list
VERBOSE [U=15702,P=1972264]addDevMount() Default mount: /dev:/dev
DEBUG [U=15702,P=1972264]addHostMount() Not mounting host file systems per configuration
VERBOSE [U=15702,P=1972264]addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=15702,P=1972264]addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=15702,P=1972264]addHomeStagingDir() Staging home directory (/home/users/lpeyrala) at /usr/local/var/singularity/mnt/session/home/users/lpeyrala
DEBUG [U=15702,P=1972264]addHomeMount() Adding home directory mount [/usr/local/var/singularity/mnt/session/home/users/lpeyrala:/home/users/lpeyrala] to list using layer: overlay
DEBUG [U=15702,P=1972264]addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=15702,P=1972264]addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=15702,P=1972264]addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=15702,P=1972264]addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=15702,P=1972264]addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libcuda.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libcuda.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libEGL_nvidia.so.0 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libEGL.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libEGL.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLdispatch.so.0 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv1_CM.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv1_CM.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv2_nvidia.so.2 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv2.so.2 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv2.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/libGL.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGL.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLX_nvidia.so.0 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLX.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLX.so.0 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvcuvid.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvcuvid.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-cbl.so.470.57.02 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-cfg.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-cfg.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-compiler.so.470.57.02 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-eglcore.so.470.57.02 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-egl-wayland.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-encode.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-encode.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-fbc.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-fbc.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-glcore.so.470.57.02 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-glsi.so.470.57.02 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.470.57.02 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-gtk2.so.470.57.02 to mount list
DEBUG [U=15702,P=1972290]startup() singularity runtime engine selected
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-gtk3.so.470.57.02 to mount list
VERBOSE [U=15702,P=1972290]startup() Serve RPC requests
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ifr.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ifr.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ml.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ml.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-opencl.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-opticalflow.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-rtcore.so.470.57.02 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-tls.so.470.57.02 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvoptix.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /usr/local/cuda-11.4/targets/x86_64-linux/lib/libOpenCL.so.1 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /usr/local/cuda-11.4/targets/x86_64-linux/lib/libOpenCL.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libOpenGL.so.0 to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libOpenGL.so to mount list
DEBUG [U=15702,P=1972264]addLibsMount() Add library /lib/x86_64-linux-gnu/libvdpau_nvidia.so to mount list
DEBUG [U=15702,P=1972264]addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=15702,P=1972264]addFilesMount() Adding file /usr/bin/nvidia-smi:/usr/bin/nvidia-smi to mount list
DEBUG [U=15702,P=1972264]addFilesMount() Adding file /usr/bin/nvidia-debugdump:/usr/bin/nvidia-debugdump to mount list
DEBUG [U=15702,P=1972264]addFilesMount() Adding file /usr/bin/nvidia-persistenced:/usr/bin/nvidia-persistenced to mount list
DEBUG [U=15702,P=1972264]addFilesMount() Adding file /usr/bin/nvidia-cuda-mps-control:/usr/bin/nvidia-cuda-mps-control to mount list
DEBUG [U=15702,P=1972264]addFilesMount() Adding file /usr/bin/nvidia-cuda-mps-server:/usr/bin/nvidia-cuda-mps-server to mount list
DEBUG [U=15702,P=1972264]addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=15702,P=1972264]addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=15702,P=1972264]addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=15702,P=1972264]create() Mount all
DEBUG [U=15702,P=1972264]mountGeneric() Mounting tmpfs to /usr/local/var/singularity/mnt/session
DEBUG [U=15702,P=1972264]mountImage() Mounting loop device /dev/loop17 to /usr/local/var/singularity/mnt/session/rootfs of type squashfs
DEBUG [U=15702,P=1972264]mountGeneric() Mounting overlay to /usr/local/var/singularity/mnt/session/final
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final
DEBUG [U=15702,P=1972264]setPropagationMount() Set RPC mount propagation flag to SLAVE
VERBOSE [U=15702,P=1972264]Passwd() Checking for template passwd file: /usr/local/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=15702,P=1972264]Passwd() Creating passwd content
VERBOSE [U=15702,P=1972264]Passwd() Creating template passwd file and appending user data: /usr/local/var/singularity/mnt/session/rootfs/etc/passwd
DEBUG [U=15702,P=1972264]addIdentityMount() Adding /etc/passwd to mount list
VERBOSE [U=15702,P=1972264]addIdentityMount() Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=15702,P=1972264]Group() Checking for template group file: /usr/local/var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=15702,P=1972264]Group() Creating group content
DEBUG [U=15702,P=1972264]addIdentityMount() Adding /etc/group to mount list
VERBOSE [U=15702,P=1972264]addIdentityMount() Default mount: /etc/group:/etc/group
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /dev to /usr/local/var/singularity/mnt/session/final/dev
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /etc/localtime to /usr/local/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /etc/hosts to /usr/local/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /proc to /usr/local/var/singularity/mnt/session/final/proc
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/proc
DEBUG [U=15702,P=1972264]mountGeneric() Mounting sysfs to /usr/local/var/singularity/mnt/session/final/sys
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /home/users/lpeyrala to /usr/local/var/singularity/mnt/session/home/users/lpeyrala
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/home/users/lpeyrala
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/home/users/lpeyrala to /usr/local/var/singularity/mnt/session/final/home/users/lpeyrala
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /tmp to /usr/local/var/singularity/mnt/session/final/tmp
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/tmp
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /var/tmp to /usr/local/var/singularity/mnt/session/final/var/tmp
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/var/tmp
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libcuda.so to /usr/local/var/singularity/mnt/session/libs/libcuda.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libcuda.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libcuda.so.1 to /usr/local/var/singularity/mnt/session/libs/libcuda.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libcuda.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libEGL_nvidia.so.0 to /usr/local/var/singularity/mnt/session/libs/libEGL_nvidia.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libEGL_nvidia.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libEGL.so to /usr/local/var/singularity/mnt/session/libs/libEGL.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libEGL.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libEGL.so.1 to /usr/local/var/singularity/mnt/session/libs/libEGL.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libEGL.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLdispatch.so.0 to /usr/local/var/singularity/mnt/session/libs/libGLdispatch.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLdispatch.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.1 to /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM_nvidia.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM_nvidia.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv1_CM.so.1 to /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv1_CM.so to /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv2_nvidia.so.2 to /usr/local/var/singularity/mnt/session/libs/libGLESv2_nvidia.so.2
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv2_nvidia.so.2
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv2.so.2 to /usr/local/var/singularity/mnt/session/libs/libGLESv2.so.2
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv2.so.2
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv2.so to /usr/local/var/singularity/mnt/session/libs/libGLESv2.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv2.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/libGL.so.1 to /usr/local/var/singularity/mnt/session/libs/libGL.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGL.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGL.so to /usr/local/var/singularity/mnt/session/libs/libGL.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGL.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLX_nvidia.so.0 to /usr/local/var/singularity/mnt/session/libs/libGLX_nvidia.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLX_nvidia.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLX.so to /usr/local/var/singularity/mnt/session/libs/libGLX.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLX.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLX.so.0 to /usr/local/var/singularity/mnt/session/libs/libGLX.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLX.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvcuvid.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvcuvid.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvcuvid.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvcuvid.so to /usr/local/var/singularity/mnt/session/libs/libnvcuvid.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvcuvid.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-cbl.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-cbl.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-cbl.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-cfg.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-cfg.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-cfg.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-cfg.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-cfg.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-cfg.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-compiler.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-compiler.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-compiler.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-eglcore.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-eglcore.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-eglcore.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-egl-wayland.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-egl-wayland.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-egl-wayland.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-encode.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-encode.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-encode.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-encode.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-encode.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-encode.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-fbc.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-fbc.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-fbc.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-fbc.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-fbc.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-fbc.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-glcore.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-glcore.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-glcore.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-glsi.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-glsi.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-glsi.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-glvkspirv.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-glvkspirv.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-gtk2.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-gtk2.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-gtk2.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-gtk3.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-gtk3.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-gtk3.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ifr.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-ifr.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ifr.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ifr.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-ifr.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ifr.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ml.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-ml.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ml.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ml.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-ml.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ml.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-opencl.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-opencl.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-opencl.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-opticalflow.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-opticalflow.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-opticalflow.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-opticalflow.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-opticalflow.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-ptxjitcompiler.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ptxjitcompiler.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-ptxjitcompiler.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ptxjitcompiler.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-rtcore.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-rtcore.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-rtcore.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-tls.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-tls.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-tls.so.470.57.02
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvoptix.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvoptix.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvoptix.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/local/cuda-11.4/targets/x86_64-linux/lib/libOpenCL.so.1 to /usr/local/var/singularity/mnt/session/libs/libOpenCL.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libOpenCL.so.1
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/local/cuda-11.4/targets/x86_64-linux/lib/libOpenCL.so to /usr/local/var/singularity/mnt/session/libs/libOpenCL.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libOpenCL.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libOpenGL.so.0 to /usr/local/var/singularity/mnt/session/libs/libOpenGL.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libOpenGL.so.0
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libOpenGL.so to /usr/local/var/singularity/mnt/session/libs/libOpenGL.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libOpenGL.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /lib/x86_64-linux-gnu/libvdpau_nvidia.so to /usr/local/var/singularity/mnt/session/libs/libvdpau_nvidia.so
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libvdpau_nvidia.so
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/libs to /usr/local/var/singularity/mnt/session/final/.singularity.d/libs
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/.singularity.d/libs
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/bin/nvidia-smi to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-smi
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-smi
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/bin/nvidia-debugdump to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-debugdump
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-debugdump
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/bin/nvidia-persistenced to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-persistenced
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-persistenced
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/bin/nvidia-cuda-mps-control to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-cuda-mps-control
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-cuda-mps-control
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/bin/nvidia-cuda-mps-server to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-cuda-mps-server
DEBUG [U=15702,P=1972264]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-cuda-mps-server
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/etc/resolv.conf to /usr/local/var/singularity/mnt/session/final/etc/resolv.conf
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/etc/passwd to /usr/local/var/singularity/mnt/session/final/etc/passwd
DEBUG [U=15702,P=1972264]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/etc/group to /usr/local/var/singularity/mnt/session/final/etc/group
DEBUG [U=15702,P=1972264]addCwdMount() Using /home/users/lpeyrala as current working directory
VERBOSE [U=15702,P=1972264]addCwdMount() /home/users/lpeyrala found within container
DEBUG [U=15702,P=1972264]create() Chroot into /usr/local/var/singularity/mnt/session/final
DEBUG [U=15702,P=1972290]Chroot() Hold reference to host / directory
DEBUG [U=15702,P=1972290]Chroot() Called pivot_root on /usr/local/var/singularity/mnt/session/final
DEBUG [U=15702,P=1972290]Chroot() Change current directory to host / directory
DEBUG [U=15702,P=1972290]Chroot() Apply slave mount propagation for host / directory
DEBUG [U=15702,P=1972290]Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=15702,P=1972290]Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=15702,P=1972264]create() Chdir into / to avoid errors
VERBOSE [U=0,P=1972289] wait_child() rpc server exited with status 0
DEBUG [U=0,P=1972289] init() Set container privileges
DEBUG [U=0,P=1972289] apply_privileges() Effective capabilities: 0x0000000000000000
DEBUG [U=0,P=1972289] apply_privileges() Permitted capabilities: 0x0000000000000000
DEBUG [U=0,P=1972289] apply_privileges() Bounding capabilities: 0x0000000000000000
DEBUG [U=0,P=1972289] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=1972289] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=1972289] apply_privileges() Set user ID to 15702
DEBUG [U=15702,P=1972289] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=15702,P=1972289]startup() singularity runtime engine selected
VERBOSE [U=15702,P=1972289]startup() Execute stage 2
DEBUG [U=15702,P=1972289]StageTwo() Entering stage 2
DEBUG [U=15702,P=1972289]StartProcess() Setting umask in container to 0022
DEBUG [U=15702,P=1972289]func1() Not exporting "BASH_FUNC__module_raw%%" to container environment: invalid key
DEBUG [U=15702,P=1972289]func1() Not exporting "BASH_FUNC_module%%" to container environment: invalid key
DEBUG [U=15702,P=1972289]func1() Not exporting "BASH_FUNC_switchml%%" to container environment: invalid key
DEBUG [U=15702,P=1972289]sylogBuiltin() Sourcing /.singularity.d/env/01-base.sh
DEBUG [U=15702,P=1972289]sylogBuiltin() Sourcing /.singularity.d/env/10-docker2singularity.sh
DEBUG [U=15702,P=1972289]sylogBuiltin() Sourcing /.singularity.d/env/90-environment.sh
DEBUG [U=15702,P=1972289]sylogBuiltin() Sourcing /.singularity.d/env/94-appsbase.sh
DEBUG [U=15702,P=1972289]sylogBuiltin() Sourcing /.singularity.d/env/95-apps.sh
DEBUG [U=15702,P=1972289]sylogBuiltin() Sourcing /.singularity.d/env/99-base.sh
DEBUG [U=15702,P=1972289]sylogBuiltin() Sourcing /.singularity.d/env/99-runtimevars.sh
DEBUG [U=15702,P=1972289]sylogBuiltin() Running action command run
DEBUG [U=15702,P=1972264]PostStartProcess() Post start process
^C(base) lpeyrala@saturn:~$ singularity -d run --nv e4s-gpu-x86_64.sif
DEBUG [U=15702,P=1976468]persistentPreRun() Singularity version: 3.9.3
DEBUG [U=15702,P=1976468]persistentPreRun() Parsing configuration file /usr/local/etc/singularity/singularity.conf
DEBUG [U=15702,P=1976468]handleConfDir() /home/users/lpeyrala/.singularity already exists. Not creating.
DEBUG [U=15702,P=1976468]execStarter() Saving umask 0022 for propagation into container
DEBUG [U=15702,P=1976468]execStarter() Checking for encrypted system partition
DEBUG [U=15702,P=1976468]Init() Image format detection
DEBUG [U=15702,P=1976468]Init() Check for sandbox image format
DEBUG [U=15702,P=1976468]Init() sandbox format initializer returned: not a directory image
DEBUG [U=15702,P=1976468]Init() Check for sif image format
DEBUG [U=15702,P=1976468]Init() sif image format detected
DEBUG [U=15702,P=1976468]setNVLegacyConfig() Using legacy binds for nv GPU setup
VERBOSE [U=15702,P=1976468]NvidiaIpcsPath() persistenced socket /var/run/nvidia-persistenced/socket not found
DEBUG [U=15702,P=1976468]findFromConfigOnly() Using "ldconfig" at "/usr/sbin/ldconfig.real" (from singularity.conf)
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding CONDA_EXE environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding _CE_M environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding ENV environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding XDG_SESSION_TYPE environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding CONDA_PREFIX environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding MODULESHOME environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding MANPATH environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding MOTD_SHOWN environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding CONDA_PROMPT_MODIFIER environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding LC_TERMINAL environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding SSH_CONNECTION environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding DOCKER_CLI_EXPERIMENTAL environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding MODULEPATH_modshare environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding XDG_SESSION_CLASS environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding _CE_CONDA environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding USER environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding CONDA_SHLVL environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding LOADEDMODULES environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding LC_TERMINAL_VERSION environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding BASH_ENV environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding XDG_SESSION_ID environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding CONDA_PYTHON_EXE environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding GO111MODULE environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding SSH_CLIENT environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding CONDA_DEFAULT_ENV environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding XDG_DATA_DIRS environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding MODULEPATH environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding DBUS_SESSION_BUS_ADDRESS environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding SSH_TTY environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding MODULES_CMD environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding BASH_FUNC_switchml%% environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding BASH_FUNC_module%% environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding BASH_FUNC__module_raw%% environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding _ environment variable
DEBUG [U=15702,P=1976468]SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=15702,P=1976468]SetContainerEnv() Setting HOME=/home/users/lpeyrala
VERBOSE [U=15702,P=1976468]SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=15702,P=1976468]init() Use starter binary /usr/local/libexec/singularity/bin/starter-suid
DEBUG [U=15702,P=1976468]Exec() Setting GOGC=off for starter
VERBOSE [U=0,P=1976468] print() Set messagelevel to: 5
VERBOSE [U=0,P=1976468] init() Starter initialization
DEBUG [U=0,P=1976468] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=1976468] load_overlay_module() Overlay seems supported by the kernel
VERBOSE [U=0,P=1976468] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=1976468] priv_drop() Drop root privileges
DEBUG [U=15702,P=1976468] read_engine_config() Read engine configuration
DEBUG [U=15702,P=1976468] cleanenv() Clearing env var LOAD_OVERLAY_MODULE=1
DEBUG [U=15702,P=1976468] cleanenv() Keeping env var SINGULARITY_MESSAGELEVEL=5
DEBUG [U=15702,P=1976468] cleanenv() Clearing env var ENGINE_CONFIG1={"engineName":"singularity","containerID":"","engineConfig":{"jsonConfig":{"filesPath":["/usr/bin/nvidia-smi:/usr/bin/nvidia-smi","/usr/bin/nvidia-debugdump:/usr/bin/nvidia-debugdump","/usr/bin/nvidia-persistenced:/usr/bin/nvidia-persistenced","/usr/bin/nvidia-cuda-mps-control:/usr/bin/nvidia-cuda-mps-control","/usr/bin/nvidia-cuda-mps-server:/usr/bin/nvidia-cuda-mps-server"],"librariesPath":["/lib/x86_64-linux-gnu/libcuda.so.1","/lib/x86_64-linux-gnu/libcuda.(TRUNCATED...)DEBUG [U=15702,P=1976468] cleanenv() Clearing env var ENGINE_CONFIG_CHUNKS=1
DEBUG [U=15702,P=1976468] cleanenv() Keeping env var GOGC=off
DEBUG [U=15702,P=1976468] init() Wait completion of stage1
VERBOSE [U=15702,P=1976500] priv_drop() Drop root privileges permanently
DEBUG [U=15702,P=1976500] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=15702,P=1976500] init() Spawn stage 1
DEBUG [U=15702,P=1976500]startup() singularity runtime engine selected
VERBOSE [U=15702,P=1976500]startup() Execute stage 1
DEBUG [U=15702,P=1976500]StageOne() Entering stage 1
DEBUG [U=15702,P=1976500]prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=15702,P=1976500]prepareAutofs() Found "/storage" as autofs mount point
DEBUG [U=15702,P=1976500]prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=15702,P=1976500]prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=15702,P=1976500]keepAutofsMount() Open file descriptor for /storage/users/lpeyrala
DEBUG [U=15702,P=1976500]keepAutofsMount() Open file descriptor for /storage/users/lpeyrala
DEBUG [U=15702,P=1976500]Init() Image format detection
DEBUG [U=15702,P=1976500]Init() Check for sandbox image format
DEBUG [U=15702,P=1976500]Init() sandbox format initializer returned: not a directory image
DEBUG [U=15702,P=1976500]Init() Check for sif image format
DEBUG [U=15702,P=1976500]Init() sif image format detected
DEBUG [U=15702,P=1976500]setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=15702,P=1976500]setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=15702,P=1976468] wait_child() stage 1 exited with status 0
DEBUG [U=15702,P=1976468] cleanup_fd() Setting FD_CLOEXEC on starter fd 3
DEBUG [U=15702,P=1976468] cleanup_fd() Close file descriptor 4
DEBUG [U=15702,P=1976468] cleanup_fd() Close file descriptor 5
DEBUG [U=15702,P=1976468] cleanup_fd() Close file descriptor 6
DEBUG [U=15702,P=1976468] cleanup_fd() Close file descriptor 7
DEBUG [U=15702,P=1976468] cleanup_fd() Setting FD_CLOEXEC on starter fd 8
DEBUG [U=15702,P=1976468] cleanup_fd() Close file descriptor 9
DEBUG [U=15702,P=1976468] cleanup_fd() Setting FD_CLOEXEC on starter fd 10
DEBUG [U=15702,P=1976468] init() Set child signal mask
DEBUG [U=15702,P=1976468] init() Create socketpair for master communication channel
DEBUG [U=15702,P=1976468] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=15702,P=1976468] priv_escalate() Get root privileges
VERBOSE [U=0,P=1976468] priv_escalate() Change filesystem uid to 15702
VERBOSE [U=0,P=1976468] init() Spawn master process
DEBUG [U=0,P=1976506] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=1976506] create_namespace() Create mount namespace
VERBOSE [U=0,P=1976468] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=1976468] enter_namespace() Opening namespace file ns/mnt
DEBUG [U=0,P=1976468] set_master_privileges() Set master privileges
VERBOSE [U=0,P=1976506] create_namespace() Create mount namespace
DEBUG [U=0,P=1976468] apply_privileges() Effective capabilities: 0x00000000000000c0
DEBUG [U=0,P=1976468] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=1976468] apply_privileges() Bounding capabilities: 0x0000003fffffffff
DEBUG [U=0,P=1976468] apply_privileges() Inheritable capabilities: 0x0000003fffffffff
DEBUG [U=0,P=1976468] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=1976468] apply_privileges() Set user ID to 15702
DEBUG [U=0,P=1976507] set_rpc_privileges() Set RPC privileges
DEBUG [U=0,P=1976507] apply_privileges() Effective capabilities: 0x0000000000200000
DEBUG [U=0,P=1976507] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=1976507] apply_privileges() Bounding capabilities: 0x0000000008204000
DEBUG [U=0,P=1976507] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=1976507] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=1976507] apply_privileges() Set user ID to 15702
DEBUG [U=15702,P=1976507] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=15702,P=1976507] init() Spawn RPC server
DEBUG [U=15702,P=1976468]startup() singularity runtime engine selected
VERBOSE [U=15702,P=1976468]startup() Execute master process
DEBUG [U=15702,P=1976468]setupSessionLayout() Using Layer system: overlay
DEBUG [U=15702,P=1976468]setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=15702,P=1976468]addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=15702,P=1976468]addRootfsMount() Image type is 4096
DEBUG [U=15702,P=1976468]addRootfsMount() Mounting block [squashfs] image: /home/users/lpeyrala/e4s-gpu-x86_64.sif
DEBUG [U=15702,P=1976468]addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=15702,P=1976468]addKernelMount() Adding proc to mount list
VERBOSE [U=15702,P=1976468]addKernelMount() Default mount: /proc:/proc
DEBUG [U=15702,P=1976468]addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=15702,P=1976468]addKernelMount() Adding sysfs to mount list
VERBOSE [U=15702,P=1976468]addKernelMount() Default mount: /sys:/sys
DEBUG [U=15702,P=1976468]addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=15702,P=1976468]addDevMount() Adding dev to mount list
VERBOSE [U=15702,P=1976468]addDevMount() Default mount: /dev:/dev
DEBUG [U=15702,P=1976468]addHostMount() Not mounting host file systems per configuration
VERBOSE [U=15702,P=1976468]addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=15702,P=1976468]addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=15702,P=1976468]addHomeStagingDir() Staging home directory (/home/users/lpeyrala) at /usr/local/var/singularity/mnt/session/home/users/lpeyrala
DEBUG [U=15702,P=1976468]addHomeMount() Adding home directory mount [/usr/local/var/singularity/mnt/session/home/users/lpeyrala:/home/users/lpeyrala] to list using layer: overlay
DEBUG [U=15702,P=1976468]addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=15702,P=1976468]addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=15702,P=1976468]addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=15702,P=1976468]addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=15702,P=1976468]addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libcuda.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libcuda.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libEGL_nvidia.so.0 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libEGL.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libEGL.so.1 to mount list
DEBUG [U=15702,P=1976507]startup() singularity runtime engine selected
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLdispatch.so.0 to mount list
VERBOSE [U=15702,P=1976507]startup() Serve RPC requests
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv1_CM.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv1_CM.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv2_nvidia.so.2 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv2.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLESv2.so.2 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/libGL.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGL.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLX_nvidia.so.0 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLX.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libGLX.so.0 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvcuvid.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvcuvid.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-cbl.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-cfg.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-cfg.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-compiler.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-eglcore.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-egl-wayland.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-encode.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-encode.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-fbc.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-fbc.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-glcore.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-glsi.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-gtk2.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-gtk3.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ifr.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ifr.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ml.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ml.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-opencl.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-opticalflow.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-rtcore.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvidia-tls.so.470.57.02 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libnvoptix.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /usr/local/cuda-11.4/targets/x86_64-linux/lib/libOpenCL.so.1 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libOpenCL.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libOpenGL.so to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libOpenGL.so.0 to mount list
DEBUG [U=15702,P=1976468]addLibsMount() Add library /lib/x86_64-linux-gnu/libvdpau_nvidia.so to mount list
DEBUG [U=15702,P=1976468]addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=15702,P=1976468]addFilesMount() Adding file /usr/bin/nvidia-smi:/usr/bin/nvidia-smi to mount list
DEBUG [U=15702,P=1976468]addFilesMount() Adding file /usr/bin/nvidia-debugdump:/usr/bin/nvidia-debugdump to mount list
DEBUG [U=15702,P=1976468]addFilesMount() Adding file /usr/bin/nvidia-persistenced:/usr/bin/nvidia-persistenced to mount list
DEBUG [U=15702,P=1976468]addFilesMount() Adding file /usr/bin/nvidia-cuda-mps-control:/usr/bin/nvidia-cuda-mps-control to mount list
DEBUG [U=15702,P=1976468]addFilesMount() Adding file /usr/bin/nvidia-cuda-mps-server:/usr/bin/nvidia-cuda-mps-server to mount list
DEBUG [U=15702,P=1976468]addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=15702,P=1976468]addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=15702,P=1976468]addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=15702,P=1976468]create() Mount all
DEBUG [U=15702,P=1976468]mountGeneric() Mounting tmpfs to /usr/local/var/singularity/mnt/session
DEBUG [U=15702,P=1976468]mountImage() Mounting loop device /dev/loop17 to /usr/local/var/singularity/mnt/session/rootfs of type squashfs
DEBUG [U=15702,P=1976468]mountGeneric() Mounting overlay to /usr/local/var/singularity/mnt/session/final
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final
DEBUG [U=15702,P=1976468]setPropagationMount() Set RPC mount propagation flag to SLAVE
VERBOSE [U=15702,P=1976468]Passwd() Checking for template passwd file: /usr/local/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=15702,P=1976468]Passwd() Creating passwd content
VERBOSE [U=15702,P=1976468]Passwd() Creating template passwd file and appending user data: /usr/local/var/singularity/mnt/session/rootfs/etc/passwd
DEBUG [U=15702,P=1976468]addIdentityMount() Adding /etc/passwd to mount list
VERBOSE [U=15702,P=1976468]addIdentityMount() Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=15702,P=1976468]Group() Checking for template group file: /usr/local/var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=15702,P=1976468]Group() Creating group content
DEBUG [U=15702,P=1976468]addIdentityMount() Adding /etc/group to mount list
VERBOSE [U=15702,P=1976468]addIdentityMount() Default mount: /etc/group:/etc/group
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /dev to /usr/local/var/singularity/mnt/session/final/dev
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /etc/localtime to /usr/local/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /etc/hosts to /usr/local/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /proc to /usr/local/var/singularity/mnt/session/final/proc
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/proc
DEBUG [U=15702,P=1976468]mountGeneric() Mounting sysfs to /usr/local/var/singularity/mnt/session/final/sys
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /home/users/lpeyrala to /usr/local/var/singularity/mnt/session/home/users/lpeyrala
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/home/users/lpeyrala
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/home/users/lpeyrala to /usr/local/var/singularity/mnt/session/final/home/users/lpeyrala
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /tmp to /usr/local/var/singularity/mnt/session/final/tmp
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/tmp
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /var/tmp to /usr/local/var/singularity/mnt/session/final/var/tmp
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/var/tmp
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libcuda.so.1 to /usr/local/var/singularity/mnt/session/libs/libcuda.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libcuda.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libcuda.so to /usr/local/var/singularity/mnt/session/libs/libcuda.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libcuda.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libEGL_nvidia.so.0 to /usr/local/var/singularity/mnt/session/libs/libEGL_nvidia.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libEGL_nvidia.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libEGL.so to /usr/local/var/singularity/mnt/session/libs/libEGL.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libEGL.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libEGL.so.1 to /usr/local/var/singularity/mnt/session/libs/libEGL.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libEGL.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLdispatch.so.0 to /usr/local/var/singularity/mnt/session/libs/libGLdispatch.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLdispatch.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.1 to /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM_nvidia.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM_nvidia.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv1_CM.so to /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv1_CM.so.1 to /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv1_CM.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv2_nvidia.so.2 to /usr/local/var/singularity/mnt/session/libs/libGLESv2_nvidia.so.2
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv2_nvidia.so.2
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv2.so to /usr/local/var/singularity/mnt/session/libs/libGLESv2.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv2.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLESv2.so.2 to /usr/local/var/singularity/mnt/session/libs/libGLESv2.so.2
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLESv2.so.2
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/libGL.so.1 to /usr/local/var/singularity/mnt/session/libs/libGL.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGL.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGL.so to /usr/local/var/singularity/mnt/session/libs/libGL.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGL.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLX_nvidia.so.0 to /usr/local/var/singularity/mnt/session/libs/libGLX_nvidia.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLX_nvidia.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLX.so to /usr/local/var/singularity/mnt/session/libs/libGLX.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLX.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libGLX.so.0 to /usr/local/var/singularity/mnt/session/libs/libGLX.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libGLX.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvcuvid.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvcuvid.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvcuvid.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvcuvid.so to /usr/local/var/singularity/mnt/session/libs/libnvcuvid.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvcuvid.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-cbl.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-cbl.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-cbl.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-cfg.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-cfg.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-cfg.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-cfg.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-cfg.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-cfg.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-compiler.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-compiler.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-compiler.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-eglcore.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-eglcore.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-eglcore.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-egl-wayland.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-egl-wayland.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-egl-wayland.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-encode.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-encode.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-encode.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-encode.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-encode.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-encode.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-fbc.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-fbc.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-fbc.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-fbc.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-fbc.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-fbc.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-glcore.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-glcore.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-glcore.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-glsi.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-glsi.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-glsi.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-glvkspirv.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-glvkspirv.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-gtk2.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-gtk2.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-gtk2.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-gtk3.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-gtk3.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-gtk3.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ifr.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-ifr.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ifr.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ifr.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-ifr.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ifr.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ml.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-ml.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ml.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ml.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-ml.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ml.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-opencl.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-opencl.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-opencl.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-opticalflow.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-opticalflow.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-opticalflow.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-opticalflow.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-opticalflow.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvidia-ptxjitcompiler.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ptxjitcompiler.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so to /usr/local/var/singularity/mnt/session/libs/libnvidia-ptxjitcompiler.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-ptxjitcompiler.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-rtcore.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-rtcore.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-rtcore.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvidia-tls.so.470.57.02 to /usr/local/var/singularity/mnt/session/libs/libnvidia-tls.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvidia-tls.so.470.57.02
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libnvoptix.so.1 to /usr/local/var/singularity/mnt/session/libs/libnvoptix.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libnvoptix.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/local/cuda-11.4/targets/x86_64-linux/lib/libOpenCL.so.1 to /usr/local/var/singularity/mnt/session/libs/libOpenCL.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libOpenCL.so.1
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libOpenCL.so to /usr/local/var/singularity/mnt/session/libs/libOpenCL.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libOpenCL.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libOpenGL.so to /usr/local/var/singularity/mnt/session/libs/libOpenGL.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libOpenGL.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libOpenGL.so.0 to /usr/local/var/singularity/mnt/session/libs/libOpenGL.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libOpenGL.so.0
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /lib/x86_64-linux-gnu/libvdpau_nvidia.so to /usr/local/var/singularity/mnt/session/libs/libvdpau_nvidia.so
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/libs/libvdpau_nvidia.so
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/libs to /usr/local/var/singularity/mnt/session/final/.singularity.d/libs
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/.singularity.d/libs
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/bin/nvidia-smi to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-smi
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-smi
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/bin/nvidia-debugdump to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-debugdump
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-debugdump
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/bin/nvidia-persistenced to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-persistenced
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-persistenced
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/bin/nvidia-cuda-mps-control to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-cuda-mps-control
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-cuda-mps-control
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/bin/nvidia-cuda-mps-server to /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-cuda-mps-server
DEBUG [U=15702,P=1976468]mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/bin/nvidia-cuda-mps-server
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/etc/resolv.conf to /usr/local/var/singularity/mnt/session/final/etc/resolv.conf
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/etc/passwd to /usr/local/var/singularity/mnt/session/final/etc/passwd
DEBUG [U=15702,P=1976468]mountGeneric() Mounting /usr/local/var/singularity/mnt/session/etc/group to /usr/local/var/singularity/mnt/session/final/etc/group
DEBUG [U=15702,P=1976468]addCwdMount() Using /home/users/lpeyrala as current working directory
VERBOSE [U=15702,P=1976468]addCwdMount() /home/users/lpeyrala found within container
DEBUG [U=15702,P=1976468]create() Chroot into /usr/local/var/singularity/mnt/session/final
DEBUG [U=15702,P=1976507]Chroot() Hold reference to host / directory
DEBUG [U=15702,P=1976507]Chroot() Called pivot_root on /usr/local/var/singularity/mnt/session/final
DEBUG [U=15702,P=1976507]Chroot() Change current directory to host / directory
DEBUG [U=15702,P=1976507]Chroot() Apply slave mount propagation for host / directory
DEBUG [U=15702,P=1976507]Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=15702,P=1976507]Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=15702,P=1976468]create() Chdir into / to avoid errors
VERBOSE [U=0,P=1976506] wait_child() rpc server exited with status 0
DEBUG [U=0,P=1976506] init() Set container privileges
DEBUG [U=0,P=1976506] apply_privileges() Effective capabilities: 0x0000000000000000
DEBUG [U=0,P=1976506] apply_privileges() Permitted capabilities: 0x0000000000000000
DEBUG [U=0,P=1976506] apply_privileges() Bounding capabilities: 0x0000000000000000
DEBUG [U=0,P=1976506] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=1976506] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=1976506] apply_privileges() Set user ID to 15702
DEBUG [U=15702,P=1976506] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=15702,P=1976506]startup() singularity runtime engine selected
VERBOSE [U=15702,P=1976506]startup() Execute stage 2
DEBUG [U=15702,P=1976506]StageTwo() Entering stage 2
DEBUG [U=15702,P=1976506]StartProcess() Setting umask in container to 0022
DEBUG [U=15702,P=1976506]func1() Not exporting "BASH_FUNC__module_raw%%" to container environment: invalid key
DEBUG [U=15702,P=1976506]func1() Not exporting "BASH_FUNC_module%%" to container environment: invalid key
DEBUG [U=15702,P=1976506]func1() Not exporting "BASH_FUNC_switchml%%" to container environment: invalid key
DEBUG [U=15702,P=1976506]sylogBuiltin() Sourcing /.singularity.d/env/01-base.sh
DEBUG [U=15702,P=1976506]sylogBuiltin() Sourcing /.singularity.d/env/10-docker2singularity.sh
DEBUG [U=15702,P=1976506]sylogBuiltin() Sourcing /.singularity.d/env/90-environment.sh
DEBUG [U=15702,P=1976506]sylogBuiltin() Sourcing /.singularity.d/env/94-appsbase.sh
DEBUG [U=15702,P=1976506]sylogBuiltin() Sourcing /.singularity.d/env/95-apps.sh
DEBUG [U=15702,P=1976506]sylogBuiltin() Sourcing /.singularity.d/env/99-base.sh
DEBUG [U=15702,P=1976506]sylogBuiltin() Sourcing /.singularity.d/env/99-runtimevars.sh
DEBUG [U=15702,P=1976506]sylogBuiltin() Running action command run
DEBUG [U=15702,P=1976468]PostStartProcess() Post start process
Output of ldconfig -p | grep libnvidia on host
libnvidia-tls.so.470.57.02 (libc6,x86-64, OS ABI: Linux 2.3.99) => /lib/x86_64-linux-gnu/libnvidia-tls.so.470.57.02
libnvidia-tls.so.470.57.02 (libc6, OS ABI: Linux 2.3.99) => /lib32/libnvidia-tls.so.470.57.02
libnvidia-rtcore.so.470.57.02 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-rtcore.so.470.57.02
libnvidia-ptxjitcompiler.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1
libnvidia-ptxjitcompiler.so.1 (libc6) => /lib32/libnvidia-ptxjitcompiler.so.1
libnvidia-ptxjitcompiler.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so
libnvidia-ptxjitcompiler.so (libc6) => /lib32/libnvidia-ptxjitcompiler.so
libnvidia-opticalflow.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1
libnvidia-opticalflow.so.1 (libc6) => /lib32/libnvidia-opticalflow.so.1
libnvidia-opticalflow.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-opticalflow.so
libnvidia-opticalflow.so (libc6) => /lib32/libnvidia-opticalflow.so
libnvidia-opencl.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-opencl.so.1
libnvidia-opencl.so.1 (libc6) => /lib32/libnvidia-opencl.so.1
libnvidia-ngx.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-ngx.so.1
libnvidia-ml.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-ml.so.1
libnvidia-ml.so.1 (libc6) => /lib32/libnvidia-ml.so.1
libnvidia-ml.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-ml.so
libnvidia-ml.so (libc6) => /lib32/libnvidia-ml.so
libnvidia-ifr.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-ifr.so.1
libnvidia-ifr.so.1 (libc6) => /lib32/libnvidia-ifr.so.1
libnvidia-ifr.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-ifr.so
libnvidia-ifr.so (libc6) => /lib32/libnvidia-ifr.so
libnvidia-gtk3.so.470.57.02 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-gtk3.so.470.57.02
libnvidia-gtk2.so.470.57.02 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-gtk2.so.470.57.02
libnvidia-glvkspirv.so.470.57.02 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.470.57.02
libnvidia-glvkspirv.so.470.57.02 (libc6) => /lib32/libnvidia-glvkspirv.so.470.57.02
libnvidia-glsi.so.470.57.02 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-glsi.so.470.57.02
libnvidia-glsi.so.470.57.02 (libc6) => /lib32/libnvidia-glsi.so.470.57.02
libnvidia-glcore.so.470.57.02 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-glcore.so.470.57.02
libnvidia-glcore.so.470.57.02 (libc6) => /lib32/libnvidia-glcore.so.470.57.02
libnvidia-fbc.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-fbc.so.1
libnvidia-fbc.so.1 (libc6) => /lib32/libnvidia-fbc.so.1
libnvidia-fbc.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-fbc.so
libnvidia-fbc.so (libc6) => /lib32/libnvidia-fbc.so
libnvidia-encode.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-encode.so.1
libnvidia-encode.so.1 (libc6) => /lib32/libnvidia-encode.so.1
libnvidia-encode.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-encode.so
libnvidia-encode.so (libc6) => /lib32/libnvidia-encode.so
libnvidia-eglcore.so.470.57.02 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-eglcore.so.470.57.02
libnvidia-eglcore.so.470.57.02 (libc6) => /lib32/libnvidia-eglcore.so.470.57.02
libnvidia-egl-wayland.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-egl-wayland.so.1
libnvidia-container.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-container.so.1
libnvidia-compiler.so.470.57.02 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-compiler.so.470.57.02
libnvidia-compiler.so.470.57.02 (libc6) => /lib32/libnvidia-compiler.so.470.57.02
libnvidia-cfg.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-cfg.so.1
libnvidia-cfg.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-cfg.so
libnvidia-cbl.so.470.57.02 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-cbl.so.470.57.02
libnvidia-allocator.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-allocator.so.1
libnvidia-allocator.so.1 (libc6) => /lib32/libnvidia-allocator.so.1
libnvidia-allocator.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libnvidia-allocator.so
libnvidia-allocator.so (libc6) => /lib32/libnvidia-allocator.so
I am awaiting confirmation from our sysadmin as to the exact name of the runfile used to install CUDA driver and toolkit. Will report back here when confirmed.
I will have to report back with a smaller .sif for reproducing.
Thanks for the additional detail. I am now somewhat confused why things aren't working... since all the libraries are being found and bound as expected. I am downloading the large container and will be able to run it in one environment in due course, but if you do get a smaller reproducer (or conversely find it's difficult to reproduce with another image) that'd be great.
Pulled the image locally onto my machine with space, and it appears to work....
$ singularity run --nv e4s-gpu-x86_64.sif
Singularity> nvidia-smi
Mon Jan 24 10:12:12 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.94 Driver Version: 470.94 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:03:00.0 N/A | N/A |
| 30% 42C P0 N/A / N/A | 402MiB / 1979MiB | N/A Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Singularity>
Noting here that with my CUDA 11.4 install I have driver 470.94 whereas report has driver 470.57.02... but 470.57.02 is a CUDA 11.x release without need for forward compatibility stuff... so this should not be an issue.
Thank you for trying this out with the big image. I know it is not convenient and we appreciate your quick response.
I confirmed with our sysadmin that CUDA driver and runtime were installed via the runfile titled:
-
cuda_11.4.1_470.57.02_linux.run
I am mystified as to what is different about our system environment. In light of the fact we can use --nv --nvccli to get desired behavior, I'm unsure whether there is any reason we would need the legacy --nv option without --nvccli
Thanks for the further details. I'll try it out with exactly that driver package when I get a chance, to see if I can replicate. Since it hasn't been reported by anyone else, and there's a workaround I'll make it a lesser priority for now.
An update on this. I couldn't replicate with the version of the driver reported here, on CE 3.9.5 before I just released it. Still not sure of the cause.
Not sure it's the same issue, but I'll mention it anyway. I had problems using an A100 under Singularity for an image that worked for K80 and V100. It turned out that some inner code (not sure whether Singularity or application code) would not work if the A100 was not in MIG mode. Enabling that got it going.
Thanks for the note @michaelkarlcoleman
I'm wondering now whether the original issue is related to https://stackoverflow.com/questions/69115621/cuda-error-not-initialized-on-a100-after-server-reset
In the first post here where the issue was reported, the host nvidia-smi output is showing MIG M. .. but there are no MIG devices listed. I'm not sure whether I can reproduce exactly since the A100 I have access to are in a virtualized cloud environment..
Helol. I'm going to close this thread as there wasn't any follow-up for some time. If there's anything else to look into, please don't hesitate to re-open it.
I'm seeing the something similar. nvidia-smi on the host returns the correct output. nividia-smi through singularity reports no devices found.
(base) [adam.amster@n280 ~]$ singularity --version
singularity version 3.7.0-1.el7
Using singularity pull --dir /allen/aibs/informatics/aamster/ docker://tensorflow/tensorflow:latest-gpu
On host
(base) [adam.amster@n285 ~]$ nvidia-smi
Wed Feb 8 06:36:58 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-PCI... Off | 00000000:64:00.0 Off | 0 |
| N/A 32C P0 36W / 250W | 0MiB / 40960MiB | 3% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
singularity:
(base) [adam.amster@n285 ~]$ singularity run --nv /allen/aibs/informatics/aamster/tensorflow_latest-gpu.sif
________ _______________
___ __/__________________________________ ____/__ /________ __
__ / _ _ \_ __ \_ ___/ __ \_ ___/_ /_ __ /_ __ \_ | /| / /
_ / / __/ / / /(__ )/ /_/ / / _ __/ _ / / /_/ /_ |/ |/ /
/_/ \___//_/ /_//____/ \____//_/ /_/ /_/ \____/____/|__/
You are running this container as user with ID 20191 and group 10513,
which should map to the ID and group for your user on the Docker host. Great!
Singularity> nvidia-smi
No devices were found
@aamster - are you able to reproduce on a newer version of SingularityCE?
The version you have listed... 3.7.0 .. is prior to the fork between SingularityCE & Apptainer, so it's quite old. It definitely pre-dates the CUDA version you are using by some time.
@dtrudg unfortunately, no, since I'm using an on-prem cluster without sudo privelages, so I can't install a newer version. But I will see if it's possible. Though, the CUDA version on the machine is 11.1, not 11.7 like it says on nvidia-smi. And also the tensorflow-latest-gpu installs CUDA 11.2 I believe.
the CUDA version on the machine is 11.1, not 11.7 like it says on nvidia-smi. And also the tensorflow-latest-gpu installs CUDA 11.2 I believe
This is a bit confusing tbh... singularity --nv must be finding CUDA 11.7 libraries somewhere on the host to bind in things that get 11.7 in nvidia-smi output in the container.
I'd highly recommend requesting an update of singularity if possible. v3.9.0 also introduces an --nvccli option for --nv... which wil use NVIDIA's own tooling to bind in devices / libraries to the container. This has a lot more logic in it, and can overcome some issues.
I'm afraid I can't provide further support for 3.7.0 here, and I do run the tensorflow gpu container with new releases without issue. I'd suggest that the Slack channel or Google Group might be useful to see if someone can replicate, or has come across this in the past on the old version.
Links at - https://sylabs.io/singularity/