toil
toil copied to clipboard
DataBiosphere
Implement FUSE on Toil for privileged environments
Ex: on AWS where we have root permissions, we can install FUSE on the cluster.
┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1506
--privileged might need to be enabled. CAP_SYS_ADMIN results in a weird permission error:
ERROR [U=0,P=596] apply_privileges() Requesting capability set 0x000001ffffffffff while permitted capability set is 0x00000000a82c25ff
I assume it's supposed to be a bit mask of all the Linux capabilities, but I'm not sure where the right-side number is coming from, and what exactly controls it.
Without SYS_ADMIN, unshare is necessary to get FUSE mounts to work, but weird things happen with actually running singularity. Loop devices also will need to be created inside the container with CAP_SYS_MKNOD.
➤ Adam Novak commented:
We’re actually going to do this without --privileged but with the very latest Singularity that fixes some unprivileged-user user namespace trouble we ran into trying to get the container to run in unshare, which lets us make do with fewer permissions.
I think stxue knows how to grant just those permissions in Docker; we still need the logic in Kubernetes to do the permission grant.
There’s also going to be some loop device management glue involved, because mounting with FUSE needs a loop device free.
I was able to avoid loop devices by mounting /dev/fuse instead of /lib/modules, but getting FUSE to work in an unprivileged cluster is likely impossible for now #4807
➤ Adam Novak commented:
We need ARM builds of Singularity 4.1.2+ to ship this. We have code that works for this on the amd64 side. We’re waiting on Debian to ship some debs of this, since the SIngularity project itself only ships amd64 debs.