research: speed up search by splitting checks into separate jobs

[foresterre]

Can we speed up the search?

Currently we assume the majority of the time is spent by the Rust compiler. While this is true, the Rust compiler isn't always taking up all available performance, sometimes because it can't, sometimes because that would require further optimizations in cargo or rustc.

Hypothesis

We can use the performance not in use by the Rust compiler or cargo to start a new check

Considerations

• Checks which run ahead of time should not steal performance of the primary build
• When a primary build is finished, the next build should be prioritized as the primary build
• When, if at all, do many separate check commands which run on a single core or thread become faster than when running prioritized checks with adjacent sipping checks?

Measurements

Impl ideas

• Refactor to job-server based implementation
• Make API where child servers or processes run the actual check, while the parent process acts as orchestrator.