addHomeMount causes fatal container creation error, with non-setuid and winbindd

[SamStudio8]

Version of Singularity

$ singularity --version
singularity-ce version 3.9.2

Describe the bug When running a demo container I get the error: container creation failed: failed to add as session directory: path . is not an absolute path. I have added several sylog.Infof statements to debug and have discovered this is the end result of lookupUnixId failing to get my user information to populate the User object with buildUser. The home source and dest are left unpopulated (presumably as empty strings or nil?) and this ultimately leads to a failure to mount the path and causes the container creation to abort (see below).

INFO    [U=27900,P=36117]  create()                      About to addHomeMount
INFO    [U=27900,P=36117]  addHomeMount()                addHomeMount
INFO    [U=27900,P=36117]  getHomePaths()                SAM! About to call user.CurrentOriginal
INFO    [U=27900,P=36117]  HostUID()                     SAM! HostUID
INFO    [U=27900,P=36117]  HostUID()                     SAM! HostUID set currentUID to 27900 from os.Getuid
INFO    [U=27900,P=36117]  HostUID()                     SAM! HostUID reading data: [27900 27900 1]
INFO    [U=27900,P=36117]  HostUID()                     SAM! HostUID returning uid=27900, containerID=27900
INFO    [U=27900,P=36117]  CurrentOriginal()             SAM! Calling CurrentOriginal with uid=27900
INFO    [U=27900,P=36117]  GetPwUID()                    SAM! Calling GetPwUID on uid=27900
INFO    [U=27900,P=36117]  lookupUnixUid()               SAM! In lookupUnixId with uid=27900
INFO    [U=27900,P=36117]  retryWithBuffer()             SAM! Trying func
INFO    [U=27900,P=36117]  lookupUnixUid()               SAM! In lookupUnixId with err=no such file or directory
DEBUG   [U=27900,P=36117]  getHomePaths()                SAM! Error in getHomePaths: user: lookup userid 27900: no such file or directory
INFO    [U=27900,P=36117]  addHomeStagingDir()           SAM! Attempting to addHomeStagingDir with source= dest=
VERBOSE [U=27900,P=36117]  CleanupContainer()            Removing image tempDir /tmp/rootfs-2414195454
INFO    [U=27900,P=36117]  CleanupContainer()            Cleaning up image...
FATAL   [U=27900,P=36117]  Master()                      container creation failed: failed to add  as session directory: path . is not an absolute path

Interestingly, other calls to lookupUnixId that are conducted before this step seem to work fine.

VERBOSE [U=27900,P=47256]  init()                        Spawn stage 1
DEBUG   [U=27900,P=47256]  startup()                     singularity runtime engine selected
VERBOSE [U=27900,P=47256]  startup()                     Execute stage 1
DEBUG   [U=27900,P=47256]  StageOne()                    Entering stage 1
INFO    [U=27900,P=47256]  current()                     SAM! Looking up user with current. syscall.Getuid=27900
INFO    [U=27900,P=47256]  lookupUnixUid()               SAM! In lookupUnixId with uid=27900
INFO    [U=27900,P=47256]  retryWithBuffer()             SAM! Trying func
INFO    [U=27900,P=47256]  buildUser()                   SAM! buildUser with name=%!s(*user._Ctype_char=0x7f01000008c0), uid=%!s(user._Ctype_uint=27900), dir=%!s(*user._Ctype_char=0x7f01000008d9)

To Reproduce

• Installed singularity-ce version 3.9.2 from source with --without-suid.
• singularity pull library://lolcow
• singularity --debug exec lolcow_latest.sif cowsay moo

Expected behavior There should be a cow saying moo, instead the container creation fails.

OS / Linux Distribution

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.7 LTS (Xenial Xerus)"

Installation Method source

Additional context

• Critically, my user information is not stored in passwd and is fetched through a connect to /var/run/samba/winbindd/pipe. I have tried binding nsswitch.conf and /var/run/samba/winbindd/pipe to the container with a bind path in my singularity.conf to no avail. I don't understand enough about Singularity to reason about why this later step does not behave the same as earlier calls to lookupUnixUid -- is it inside the container context at that time?
• I noticed that the call to user.CurrentOriginal can be avoided in getHomePaths if a custom home is set. Therefore the bug can be avoided entirely with --no-home or manually specifying --home. A warning is still emitted (WARNING: user: lookup userid 27900: no such file or directory) but the container execs correctly.
• I can strace -f the creation of the container if that helps with further debugging.
• This error seems to have been encountered previously: https://github.com/apptainer/singularity/issues/6242

[SamStudio8]

Should it be the case that source and dest are set to pw.Dir after user.CurrentOriginal returns an err? https://github.com/sylabs/singularity/blob/377e2786b69315edc2820dbc0a89632141450503/internal/pkg/runtime/engine/singularity/container_linux.go#L1611-L1613

[dtrudg]

There's definitely (in accordance with the other issue you linked) a problem with the combination of user information coming from winbind, and unprivileged execution of Singularity. The error is at a point where we aren't in the container, but are in a user namespace. CurrentOriginal is concerned with checking where things are outside of any namespaces and user mappings. Something is a little different for winbind - as we do know there isn't the same issue when there is e.g. an LDAP source for the user information.

It would be good to see an strace -f output, so we can identify exactly what's happening at this point. If you are able to provide that it might help without us needing to setup a winbind environment. Thanks!

[dtrudg]

Helol. I'm going to close this thread as there wasn't any follow-up for some time. If there's anything else to look into, please don't hesitate to re-open it.

[mdehollander]

Although this issue has been closed, it hasn't been fixed in singularity itself. Maybe good to know this issue has been picked up by apptainer and there has been a workaround to fix for the issue with winbind in the 1.2.0 rc