/nix/store bind-mount to an NFS mount incompatible with cntr

[MichaelBurge]

Steps to reproduce

1. Install Nix
2. Mount an NFS to /media/nas: mount -t nfs nas.lan:/ /media/nas
3. Bind mount /nix/store: mount -o bind /media/nas/nix-store /nix/store
4. Follow instructions listed here to create a debug sandbox.
5. Observe that executing /.cntr/cntr-exec gives the error "Operation not supported". And nothing in nix-store or my regular user is accessible(with the same error), even if I set --effective-user to root or my regular user.
6. Unmount /nix/store, rm -rf /nix/store, and reinstall Nix
7. Observe that the same nix-build stops during buildPhase with a container whose /.cntr/cntr-exec is executable, and that the expect /nix/store tools are available and execute fine.

I speculate that two issues are relevant:

1. cntr sets CAP_SYS_CHROOT on a copy of itself, but this is likely not persisted across NFS.
2. My NFSv4 is configured to squash all user ids to the same user, but in a multi-user Nix daemon installation there are 32 nixbld1 users. So it is possible that permissions given to those are being lost during the user id squash.

It seems reasonable to throw an explicit error if the permissions responsible cannot be set, rather than build an unusable container.

[Mic92]

Usually this binary should be written to $TMPDIR and fallback to /tmp. Is $TMPDIR on your system also on NFS?

[Mic92]

I also have an experimental branch where I try to fallback to other directories: https://github.com/Mic92/cntr/pull/57/files#diff-96d5465b3dc8efadb91937f8745c3ad21bb114f524963f6ea57fa5e2ad2c435cR27

[Mic92]

Also what is the full error message?

[MichaelBurge]

1. $TMPDIR was /tmp and on a local filesystem.
2. The binary was successfully written to a directory in /tmp, and it was visible in the container ls /.cntr/ as cntr-exec
3. However, when executing /.cntr/cntr-exec within the container I get an error message /.cntr/cntr-exec: Operation not supported with no further details(it succeeds when run outside the container).
4. Additionally stat /.cntr/cntr-exec fails within the container but stat /tmp/SOME-DIRECTORY-RECENTLY-MODIFIED/cntr-exec succeeds outside the container.

[Mic92]

This looks like a problem with the Fuse rather than NFS than. What is your kernel version?

[MichaelBurge]

mburge@lam:~$ uname -a
Linux lam 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
mburge@lam:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04 LTS
Release:	20.04
Codename:	focal