[cifuzz] Execute fuzzers in parallel.

[jonathanmetzman]

Now that CIFuzz can support non-Github users we should look into parallel running of fuzzers. This will be especially useful for Skia (cc @kjlubick) which will use 64 core machines for builds. Even for github actions users this may be useful. Those machines have two core so we may be able to speed up fuzzing by 50%.

[DentonGentry]

https://buildjet.com/for-github-actions also provides drop-in replacement runners with up to 32 vCPUs.

[jonathanmetzman]

Thanks for mentioning this. Btw can you say what you use ClusterFuzzLite for? It helps me justify development and I have limited insight into who is using it right now.

[DentonGentry]

We use it in the CI for https://github.com/tailscale/tailscale, an open source VPN. We face the usual tradeoff:

• We would love to run fuzz testing even longer. It makes the code better.
• We need CI to complete quickly enough that the developer can stay on task. If it takes so long that they go work on something else while it finishes, then progress on any single project slows substantially.

We're experimenting with manually breaking our workflow into pieces using a matrix to introduce concurrency.

The other possibility is to decouple fuzzing from CI and run it periodically on a schedule at HEAD, when it can run as long as we want. The downside is turning a recently introduced problem from something which has to be fixed before the PR merges into a new weird failure which somebody needs to go look at when they have cycles.