SELinux Preventing starter-suid from Executing

[gistambo]

Greetings, We deployed singularity-ce version 3.9.2-1.el7_9 on a small Red Hat 7.9 cluster, and tried building an .sif test image from a definition file on the login node, where SELinux is in enforcing mode, like so:

$ singularity build --fakeroot test.sif test.def
ERROR  : Failed to create container process: Operation not permitted

It seems that SELinux is blocking singularity from executing, with the following error in the logs: SELinux is preventing /usr/local/singularity/3.9.2/libexec/singularity/bin/starter-suid from using the sys_admin capability.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that starter-suid should have the sys_admin capability by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'starter-suid' --raw | audit2allow -M my-startersuid#012# semodule -i my-startersuid.pp#012

Is there an SELinux-Singularity package/policy that is available for us to download, or any other solution that you can think of to avoid this issue?

Attached you will find the definition file test.def.txt

[dtrudg]

This is a little unexpected - I do my development work on a Fedora machine with selinux in enforcing mode, without issue as the binary ends up unconfined.

Does your cluster have additional hardening or non-default selinux policies applied?

What labels have been applied to the singularity starter-suid binary? I.E.

ls -lah -Z /usr/local/libexec/singularity/bin/starter-suid 
-rwsr-xr-x. 1 root root system_u:object_r:bin_t:s0 21M May 18 18:00 /usr/local/libexec/singularity/bin/starter-suid

We don't have a package that includes an SELinux policy at this time.

[gistambo]

This is a little unexpected - I do my development work on a Fedora machine with selinux in enforcing mode, without issue as the binary ends up unconfined.

Does your cluster have additional hardening or non-default selinux policies applied?

What labels have been applied to the singularity starter-suid binary? I.E.

ls -lah -Z /usr/local/libexec/singularity/bin/starter-suid 
-rwsr-xr-x. 1 root root system_u:object_r:bin_t:s0 21M May 18 18:00 /usr/local/libexec/singularity/bin/starter-suid

We don't have a package that includes an SELinux policy at this time.

That's a yes on additional hardening. Security compliance requires that we operate with SELinux confined users configured; users are mapped to user_u and admins to staff_u (with sudo configured to transition to execution under sysadm_u); note that sysadm_u has the necessary permissions because this works when we run it as root with sudo. We show the same label for starter_suid.

$ sudo ls -lahZ /usr/local/singularity/3.9.2/libexec/singularity/bin/starter-suid
-rwsr-xr-x. root root system_u:object_r:bin_t:s0       /usr/local/singularity/3.9.2/libexec/singularity/bin/starter-suid

It seems to be a restriction on capabilities under these confined users. Here are some of the specific AVC denials when we try this with SELinux in permissive mode:

type=AVC msg=audit(01/19/2022 19:53:56.439:334793) : avc: denied {​​​ dac_override }​​​ for pid=242993 comm=starter-suid capability=dac_override scontext=staff_u:staff_r:staff_t:s0 tcontext=staff_u:staff_r:staff_t:s0 tclass=capability permissive=1

type=AVC msg=audit(01/19/2022 19:53:56.440:334794) : avc: denied {​​​ setpcap }​​​ for pid=243015 comm=starter-suid capability=setpcap scontext=staff_u:staff_r:staff_t:s0 tcontext=staff_u:staff_r:staff_t:s0 tclass=capability permissive=1

[dtrudg]

Right - there's going to be quite a bit needed then, I suspect. The setuid binary is going to ask for a number of capabilities and you'll see notes around that here:

https://github.com/sylabs/singularity/blob/3edb1ec0b5cb8f7c4577227392bc7c8027590ec5/cmd/starter/c/starter.c#L358

To be honest, I think that an selinux policy that can be layered on user_u confinement is not going to be a high priority item for us on SingularityCE. We'd accept contributions, but in the open-source infrastructure we don't have a good means to test against 'true' RHEL (userland and kernel) in CI.

There's also the caveat that really no single policy is going to be appropriate. A lot of people aren't e.g. going to be interested in encryption, so a complete policy would include allowing things that are unnecessary for them.

Given the emphasis on hardening here, what are the barriers to running in non-setuid mode for your workflows? Are you able to use unprivileged user namespaces within your hardening stance?

Pinging @tri-adam in case he has any thoughts here :-)

[tri-adam]

It would seem to me that the resulting work here would ideally be specific to your organization's use of SingularityCE @gistambo, as @dtrudg suggests. We'd certainly consider a contribution to this effect if it could be made general enough to be applicable to others, and we could find a reasonable path forward for maintaining it.

Finally, I'll mention that if you're looking for help developing a policy that fits your exact organization's specific needs, Sylabs may be able to help. This space is focussed on the SingularityCE open source project, so if that's what you're looking for, it would be best to reach out to Sylabs directly (link).

[gistambo]

Given the emphasis on hardening here, what are the barriers to running in non-setuid mode for your workflows? Are you able to use unprivileged user namespaces within your hardening stance?

@dtrudg According to the documentation,

Running Singularity in non-setuid mode requires unprivileged user namespace support in the operating system kernel and does not support all features, most notably direct mounts of SIF images. This impacts integrity/security guarantees of containers at runtime.

I ignored this section while installing Singularity as I know neither what features will not be supported, nor their criticality to our HPC users.

Finally, I'll mention that if you're looking for help developing a policy that fits your exact organization's specific needs, Sylabs may be able to help. This space is focussed on the SingularityCE open source project, so if that's what you're looking for, it would be best to reach out to Sylabs directly (link).

I don't think that this would classify as a bug, so I doubt that sylabs will take an interest in this issue without us being a paying customer. Our staff would consider purchasing support, but this is a case where we must first experiment with the container platform and most importantly get user feedback and usage before procuring the Pro or Enterprise version.

Thank you @dtrudg and @tri-adam for lending us your expertise and time on this issue. I will discuss the rest of your post with our security specialist, to find out if they can make progress that we can share in this thread.

[dtrudg]

Helol. I'm going to close this thread as there wasn't any follow-up for some time. If there's anything else to look into, please don't hesitate to re-open it.