SIGSEV (signal 11 error) with Go 1.22.0 and Ubuntu 20.04 / Debian 10

[gregorex333]

Version of Singularity

Singularity-CE 4.1.0 (also tried CE 4.1.1 with same result)

Describe the bug

Fails to mount tempfs or ramfs for "build --sandbox" mainly with this error: signal number 11

"VERBOSE [U=0,P=1] wait_child() rpc server interrupted by signal number 11 FATAL [U=0,P=9662] Master() container creation failed: mount tmpfs->/usr/local/var/singularity/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /usr/local/var/singularity/mnt/session: read unix @->@: read: connection reset by peer : exit status 255"

To Reproduce

Reset my dual-boot Linux harddrive back to its initial conditions for Ubuntu focal 20.04.6 LTS installed a small number of programs needed for using the image to be built from my custom definition full list of commands / packages in attached file: full_reproduction.odt

Install GO 1.22.0 Install dependencies listed at (https://docs.sylabs.io/guides/main/admin-guide/installation.html) for Ubuntu Install singularity into usr/local with:

./mconfig &&
make -C ./builddir &&
sudo make -C ./builddir install

Run build -- sandbox on my custom definition file with a library for ubuntu:20.04 as its base AS WELL AS on this basic test case library: sudo singularity -d build --sandbox ubuntu/ library://ubuntu Fails to mount with above error

Tried more by setting these env variables & with or without various config file settings listed With or Without these env variables export SINGULARITY_TMPDIR=/home/giovannini/sandbox/temp/tmp export SINGULARITY_CACHEDIR=/home/giovannini/sandbox/temp/cache (also tried with or without sudo -E)

(also tried with singularity_conf	mount tmp = no or yes)
(also tried with singularity_conf	mount host fs = no or yes)
(also tried with singularity_conf	sessiondir_max_size = 20480 or 64)

Also tried with "mount fs" set to tempfs and ramfs. All fail to mount at the same point.

Expected behavior

Expected to mount tempfs/ramfs to create a sandbox folder OR the .sif from my defintion

OS / Linux Distribution

Ubuntu focal 20.04.6 LTS

Installation Method

Install singularity into usr/local with: ./mconfig &&
make -C ./builddir &&
sudo make -C ./builddir install using your github release source file for 4.1.0 (and previously 4.1.1 before resetting my OS)

Additional context

mount and cat /proc/self/mountinfo and build config also in attached file: mount_build_info.odt

DEBUG Example Debug when attempting to sandbox my definition file. Same point of failure for the other basic test case. with or without these variables set export SINGULARITY_TMPDIR=/home/giovannini/sandbox/temp/tmp export SINGULARITY_CACHEDIR=/home/giovannini/sandbox/temp/cache

(also tried with singularity_conf	mount tmp = no or yes)
(also tried with singularity_conf	mount host fs = no or yes)
(also tried with singularity_conf	sessiondir_max_size = 20480 or 64)
(also tried with or without sudo -E for those env vars)

Debug Log.odt

[dtrudg]

This...

wait_child() rpc server interrupted by signal number 11

Indicates that there is a segmentation fault. Please check the output of the dmesg command and provide the error messages that are there.

I believe this is probably related to an issue reported against Go 1.22.0 - https://github.com/golang/go/issues/65625 that is affecting other container runtime projects also (incus / runc).

Please try building Singularity with Go 1.21.7 from https://go.dev/dl/ instead.

[dtrudg]

The bug itself is not in Go - it's in glibc, but this is difficult to avoid:

https://github.com/opencontainers/runc/pull/4193#issuecomment-1947700151

[gregorex333]

This...

wait_child() rpc server interrupted by signal number 11

Indicates that there is a segmentation fault. Please check the output of the dmesg command and provide the error messages that are there.

I believe this is probably related to an issue reported against Go 1.22.0 - golang/go#65625 that is affecting other container runtime projects also (incus / runc).

Please try building Singularity with Go 1.21.7 from https://go.dev/dl/ instead.

dmesg message dmesg.odt

will try building with different version of GO

[dtrudg]

@gregorex333 - thanks. I believe the dmesg output there confirms it is the same issue.

[gregorex333]

The bug itself is not in Go - it's in glibc, but this is difficult to avoid:

opencontainers/runc#4193 (comment)

You seem correct. Changing GO installation and reinstalling has allowed the build to complete with exist status 0. sudo singularity -d build --sandbox ubuntu/ library://ubuntu

VERBOSE [U=0,P=283260] Full() Build complete: /home/giovannini/sandbox/ubuntu DEBUG [U=0,P=283260] cleanUp() Cleaning up "/home/giovannini/sandbox/build-temp-1227878293/rootfs" and "/tmp/bundle-temp-2089219652" INFO [U=0,P=283260] runBuild() Build complete: ubuntu/