ducttape
ducttape copied to clipboard
Throttling a single task
It might be useful to allow a maximum number of instances of a single resource-hogging task to run at a single time.
Example: Task 1 involves a large amount of disk IO (e.g. building phrase tables) and Task 2 involves a large amount of network overhead. Running multiple instances of either task in parallel will be slower than running serially. However, having an instance of task 1 in parallel with task 2 is fine.
It would be nice to define a mechanism that allows a scheduler (possibly a ducttape-internal scheduler) to enforce this constraint.