Add experimental job server

[syu-nya]

To use the job server, enable "jobserver" feature in make.conf. It currently needs to turn off the "network-sandbox" feature.

Added new execute: bin/ejobeserver standalone job server bin/ejobwrapper.sh program wrapper

Added new variables: PORTAGE_JOBSERVER_ADDRESS PORTAGE_JOBSERVER_PORT PORTAGE_JOBSERVER_MAX_JOBS PORTAGE_JOBSERVER_MIN_FREE_MEMORY PORTAGE_JOBSERVER_MAX_SYSTEM_LOAD PORTAGE_JOBSERVER_DELAY PORTAGE_JOBSERVER_REMOTE

For details check man make.conf

Signed-off-by: Sheng Yu [email protected]

[syu-nya]

Quick start:

Enable feature jobserver and disable network-sandbox in make.conf FEATURES="jobserver -network-sandbox"

Create wrapper links like:

mkdir -p /usr/local/lib/portage/jobserver

ln -s /usr/bin/ejobwrapper.sh /usr/local/lib/portage/jobserver/gcc
ln -s /usr/bin/ejobwrapper.sh /usr/local/lib/portage/jobserver/g++
ln -s /usr/bin/ejobwrapper.sh /usr/local/lib/portage/jobserver/clang
ln -s /usr/bin/ejobwrapper.sh /usr/local/lib/portage/jobserver/clang++
ln -s /usr/bin/ejobwrapper.sh /usr/local/lib/portage/jobserver/x86_64-pc-linux-gnu-gcc
ln -s /usr/bin/ejobwrapper.sh /usr/local/lib/portage/jobserver/x86_64-pc-linux-gnu-g++
ln -s /usr/bin/ejobwrapper.sh /usr/local/lib/portage/jobserver/x86_64-pc-linux-gnu-clang
ln -s /usr/bin/ejobwrapper.sh /usr/local/lib/portage/jobserver/x86_64-pc-linux-gnu-clang++
ln -s /usr/bin/ejobwrapper.sh /usr/local/lib/portage/jobserver/rustc

Add more if you needed. But do not add tools like make, cmake, cargo, python, as they could spawn others and could run into deadlock.

System installed should go to /usr/lib/portage/jobserver.

Adjust PORTAGE_JOBSERVER_MAX_JOBS in make.conf if you want.

If you want to run the primary job server yourself and see the stats, run jobserver -s before you start portage. No root needed. More configurations check jobserver --help

Set portage run parallel jobs use --jobs X if you not already. Disable all limiters like --load-average. Make sure you had enough memory.

And its done.

If you have multiple build containers. Run jobserver -a [IP] at you host system, and set PORTAGE_JOBSERVER_ADDRESS and PORTAGE_JOBSERVER_REMOTE="true" in your containers.

[xxc3nsoredxx]

Question for the future: what would install the jobwrapper.sh symlinks? Portage install script? Portage ebuild? Individual tool ebuilds? The user manually into /usr/local?

I'm leaning towards the individual tool ebuilds, so for example, GCC installs /usr/lib/.../gcc, /usr/lib/.../$target-gcc, etc. That way only symlinks for tools that exist on the machine are created.

[syu-nya]

I think the ebuild should install the symlinks, as it should knows what it going to install.

[thesamesam]

Note that we can possibly integrate with dev-util/shadowman here with already tries to handle ccache and distcc (cc @mgorny)?

(This also answers the question of where to install to.)

[syu-nya]

New options for jobserver, -S enable sandbox. Must be run as root. The job server no longer need to disable network-sandbox for local building. This is done by created a shared network namespace between jobserver and ebuild, that is still isolated and loopback only. To verify the sandbox, run as root:

jobserver -s -S

open another root shell:

nsenter --net=/proc/PID_TO_JOBSERVER/ns/net
ip addr

You can also try to install something and see if portage will using it.

The jobserver will now drop root privileges after set up PID file and namespace.

[syu-nya]

I moved the 2 executable scripts jobserver -> ejobserver jobwrapper.sh -> ejobwrapper.sh that maybe conflicting with others. Symlinks need to be updated.