Run multiple cargo tasks in parallel

[sourcefrog]

trafficstars

Although cargo and the test framework try to use many cores, they often go through phases of being single-threaded.

The overall mutation tests would go faster if we ran multiple cargo builds in parallel.

Conceivably up to NCPUs builds could be run in parallel. But on some machines that might cause thrashing or too much memory pressure. Possibly we should also look at available memory. There should at least be an option to limit it. As a starting point perhaps NCPUs/2 is reasonable.

This requires making multiple scratch directories. However, we should not make more than the number of discovered mutants.

I think rather than starting many threads I want to have a single thread start many cargo tasks each handled as non-blocking. The work in the main process does not use much CPU.

• [ ] Factor out a CargoTask with a poll method
• [ ] And perhaps a Job that proceeds through the three phases, running cargo for each. It also has a poll and eventually returns an overall outcome.
• [ ] Determine the concurrency limit: at most the number of mutants; otherwise a value specified by --jobs or NCPUS
• [ ] Make multiple build directories
• [ ] Give the progress model a concept of multiple cargo tasks being underway
• [ ] In the lab code:
  • While there's a build dir free and still work to be done, start a new job in it
  • Sleep a fraction of a second
  • Update the progress bar for all running jobs
  • Poll all jobs to see if any have finished