cirocosta
Limit concurrency
Hey,
for some workloads where too much concurrency gets in the way in the form of too much context switching it's interesting to have limited concurrency.
The easiest approach is to go the make route (--concurrency=4 would be similar to -j4). Another route would be "per group" limited concurrency, where a given group of jobs would never have more than n jobs running at the same time.
To achieve the second one we'd need to add the concept of groups or something like that.
Maybe tagging?
If we go with 2 then 1 can be achieved via some kind of "global tagging" - all the jobs would be part of this "root" group which would have a limited concurrency. The nested groups (ANDing of a tag with another tag) would take the smaller concurrency number.
Implementation-wise that could consist of maintaining a map of locks where the keys of this map would be these labels/keys/groups such that at job execution time we could decide which lock to take to wait for.
Not sure yet 🤔
