permissions for CSI volume mounted to `exec` driver don't allow task's user

[akamensky]

Nomad version

Nomad v1.4.3 (f464aca721d222ae9c1f3df643b3c3aaa20e2da7)

Operating system and Environment details

Fedora 36 (Server)

Issue

Using CSI volume mounted in the task started using exec driver is impossible due to mount point being owned by root with 0755 permissions.

Reproduction steps

• Ceph 3 node cluster
• Nomad configured using https://docs.ceph.com/en/latest/rbd/rbd-nomad/#create-a-pool
• Volume created with RW permission as "file-system"
• Job with exec driver that mounts above volume with read_only = false

Expected Result

• Mount point is owned by the same user as the one inside the namespace (default nobody)
• OR mount point is otherwise made write-able (i.e. setting permissions)

Actual Result

• Mount point is owned by root and uses 0755 permissions. Hence default nobody user can't use it.

Note

I'd consider this a bug since there is no clear way to use CSI volumes with exec driver other than running process as root.

[akamensky]

The insecure workaround is to:

1. Change client config to allow running tasks as root (i.e. "user.denylist" = "")
2. Run prestart task as user = "root" that calls chown nobody:nobody /path/to/mount

[tgross]

Hi @akamensky! Yeah this one is a little tricky because it lets a job operator overwrite the permissions of the contents of a volume outside of the user permissions they otherwise have. We've got an unfortunately-long-open issue https://github.com/hashicorp/nomad/issues/8892 that covers this, so I'm going to close this as a duplicate. Thanks for opening it though, and if you have additional thoughts feel free to add them to #8892. Thanks!

[akamensky]

@tgross I respectfully disagree on bundling those issues together. #8892 is a feature request to provide controls for managing volume permissions in Docker driver tasks, which very well could be a reason for it to be stale/forgotten for over 2 years. This issue is a bug report about volume being mounted as a user entirely different than the one that the process is running as in exec environment.

The difference here is that in case of Docker driver tasks you may have limited options on managing external volumes and it being feature request with known workaround could be put in the ice box. In case of exec driver Nomad is in control of all aspects of isolated environment and there should be no excuse to mounting volume as wrong user. If this issue is closed as a dup of the other one, then there is no guarantee that it will be fixed whenever (if at all) the other one will be looked at (that is the other issue may get fixed for docker only).

[tgross]

I see where you're going with that, but the CSI mount is working as intended; the spec doesn't include any handling of permissions (which sucks, but the CSI spec is surprisingly underbaked on those details). And whatever we do to implement this would effectively need to be done everywhere, so it's not like it's an exec-specific solution either. Marking it as a bug doesn't do a lot for prioritization on it's own, either. But sure, I'll reopen and put it on the bug backlog.

[apollo13]

Just putting down some links here that might or might not help. Podman has similar problems with user namespaces: https://github.com/containers/podman/issues/12154

Newer kernels allow id-mapping mounts: https://lore.kernel.org/lkml/[email protected]/T/#u

[116davinder]

Hopefully, this will be fixed or documented somewhere properly.