zeek
Explore adding client integration with GNU make jobserver protocol
When compiling a parser we spawn external processes to compile generated C++ files and link them into an HLTO file. We make sure to not spawn more than nproc parallel processes, but even on powerful systems especially the compilation jobs can overwhelm the system when multiple spicyc/spicyz invocations are active at the same time, e.g., in builds of larger setups with recursive invocations of make, or when executing btest with -j. Users then work around this by specifying a small value for HILTI_JIT_PARALLELISM to control the worst case, but likely slowing down the average case.
GNU make implements a job server which can be used to control parallelism in such scenarios.[^ninja] The server maintains a limited set of "tokens" which programs invoked by make obtain in order to launch jobs. With that the top-level parallelism from make -j<N> can be respected.
We should consider adding functionality so JIT can act as a job server client, ideally using a library which implements the server interaction. If a job server is detected we would override the current behavior of using the maximum concurrency of the hardware; HILTI_JIT_PARALLELISM/HIT_JIT_SEQUENTIAL would still override this behavior.
https://github.com/zeek/spicy/blob/f06bd938a53d712cc4f148dfac53570dd6cda27e/hilti/toolchain/src/compiler/jit.cc#L506-L532
In order to make this work in BTest invocations we would need to implement support for btest to act as both a job server as well as client.
[^ninja]: Ninja is in the process of implementing support for it to be a client as well as a server itself.
