Overlayfs masking/whiteout layer

[swick]

#547 adds support for overlayfs mounts.

There are important features that it doesn't provide:

1. In some situations the user and group of the files and folders of the overlayfs mount is nobody. This prevents one from mounting further files and directories on top.
2. Adding new files and directories without creating a host tree.
3. Hiding files and directories of a layer.

To solve 1. it's possible to create a layer which has directories and files with the right user, group and permissions at the mount points. Doing so is possible right now but requires the layer to live on the host.

Some users of bwrap exec bwrap and disappear, without a daemon or anything else left running. All the cleanup of resources must be done automatically by the kernel. Creating a filesystem tree that acts as an overlayfs layer on the host is therefore not a possibility.

There are two ways I can think of how to handle this:

1. Create the layer inside the /newroot with the usual --file --dir etc. parameters, then add a new --overlay-src-inside option which allows to add the layer from /newroot to the overlayfs, then possibly mount over the layer tree to prevent modification. Example: --dir /tmp/overlay/x --overlay-src lower --overlay-src-inside /tmp/overlay --ro-overlay /mnt --tmpfs /tmp/overlay makes it possible to mount on /mnt/x no matter what the lower level contains.
2. Create a new layer in /overlay-n (not /newroot, not /oldroot) whenever there are --file or --dir arguments interspersed between --overlay-src and other overlay arguments. The --file and --dir arguments then populate the layer instead of /newroot. Example: --overlay-src lower --dir /x --ro-overlay /mnt is identical to the example in 1.

Solving 3. requires creating a character device with 0/0 device number for whiteout and setting the xattr "trusted.overlay.opaque" of a directory to "y" for an opaque directory (i.e. not merging with lower levels). We could add new arguments for both. If we go with option 1, we have to allow creating them anywhere in /newroot, whereas option 2 would allow us to create them only on overlay layers in /overlay-n. Example: --overlay-src lower --dir /x --whiteout /y --opaque-dir /z --ro-overlay /mnt

I have an implementation for option 1 here https://github.com/swick/bubblewrap/tree/wip/overlayfs but I'm open to implementing option 2 if others think that's the better approach.

e: option 2 doesn't work for creating files and directories in the lowest layer. Maybe we need separate options --overlay-file, --overlay-dir, --overlay-opaque-dir and --overlay-whiteout.

@rhendric @smcv @alexlarsson

[rusty-snake]

Solving 3. requires creating a character device with 0/0 device number for whiteout and setting the xattr "trusted.overlay.opaque" of a directory to "y" for an opaque directory (i.e. not merging with lower levels).

Setting extended attributes in the trusted namespaces requires CAP_SYS_ADMIN in the initial user namespace.

For unprivileged usage overlayfs supports userxattr parameter and falls back to user namespace. However the user namespace is only usable on regular files and directories, not on block/character device files.