Create a network namespace for each PUN

[ericfranz]

Apart from completely moving the PUN into a Docker container, which would be an interesting option to explore, adding a separate network namespace for each PUN would enable using Passenger GLS for OnDemand apps on the web host.

This is under the assumption that Passenger starts GLS applications listening on loopback and thus would not be accessible outside the network namespace.

We would use a mixture of calling unshare --net or ip netns commands along with ip link for creating a veth pair for each network. The namespaces could be named (the same name as the username of the user) or possibly anonymous (created in each PUN startup and tied to the PUN PID).

┆Issue is synchronized with this Asana task by Unito

[ericfranz]

Talked with @treydock on Slack. We will explore using systemd-nspawn --share-system --network-veth. If we take this approach, we would need to specify a list of directories and files that the container should have access to, which complicates things because the processes in the container want to make use of SCL and other dependencies like Slurm client binaries or GNU core utilities.

Benefit of using systemd-nspawn is that:

1. appears it may be easier to use than using unshare and ip link directly
3. provides mount namespace setup out of the box, so we can ensure that directories in the allowlist are the only ones the user can access, along with those directories that the apps require as dependencies (is this easy or doable ?).

Drawback is the presence of mount namespaces. We would need to expand configuration options and specify a number of directories required for the dependencies to access. I'm also not sure what directories should use --directory= vs --bind= vs --tmpfs=.

It would be nice if we could initially bypass the use of this. Perhaps using --directory=/ would do it - though then we would be back to the start where we were before on the challenge of enforcing limits on directory access - but we could revisit that later with a set of configurable bind mounts that would work.

[treydock]

I think for systemd-nspawn we may also need the --private-network flag to actually get a private network namespace.