Failure with --fakeroot and overlay onto tmpfs on Ubuntu

[DrDaveD]

At least with apptainer-1.1.6, the --fakeroot using the fakeroot command and --writable-tmpfs options together on Ubuntu can cause some strange errors regarding missing commands. These errors go away when using --overlay onto a directory on an ext4 filesystem instead of --writable-tmpfs, and come back with --overlay onto a directory in another tmpfs filesystem such as /dev/shm. They also go away when using a sandbox for a container instead of a sif file, and when using /etc/sub[ug]id fakeroot instead of the fakeroot command.

To reproduce, on Ubuntu 20.04 do:

apptainer build ubuntu.sif docker://ubuntu:20.04
apptainer exec -u --fakeroot --writable-tmpfs --ignore-subuid ubuntu.sif sh -c "apt-get update && apt-get install -y openssh-client"

This ends up with errors like

dpkg: warning: 'sh' not found in PATH or not executable
dpkg: warning: 'diff' not found in PATH or not executable
dpkg: error: 2 expected programs not found in PATH or not executable
Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin
E: Sub-process /usr/bin/dpkg returned an error code (2)

I tried comparing strace -v output for both types of overlay, and the return from the stat() system call on those supposedly missing executables look identical to me, so I don't really have any idea why it is failing.

I have also seen similar errors on Ubuntu 22.04 with an ubuntu:22.04 image and upgraded fuse-overlayfs version 1.9, but only when run along with the e2e CI test from github actions, not when running it interactively.