cylc-flow
cylc-flow copied to clipboard
Batch spawn POC
Zero-th level workaround for #5437
(We still need to optimize the problematic code, but this approach might be useful in the interim, and possibly in the long run as well).
UPDATE: the main problem (n-window computation) was fixed by #5475. But it may still be worth doing this as well.
When an output gets completed, instead of spawning all children into the task pool at once, record what needs to be spawned, and spawn them batch-wise via the main loop.
This plays well with queuing, because queues work with what they've got, so tasks can be released to run throughout the long spawning period.
The example from #5437 is quite usable on this branch, although CPU remains high till spawning is complete, and only the GUI table view is workable (and that with filtering for active tasks):
[task parameters]
m = 0..6999
[scheduling]
[[queues]]
[[[default]]]
limit = 4
[[graph]]
R1 = "a => b<m>"
[runtime]
[[a]]
script = sleep 10
[[b<m>]]
Check List
- [ ] I have read
CONTRIBUTING.md
and added my name as a Code Contributor. - [ ] Contains logically grouped changes (else tidy your branch by rebase).
- [ ] Does not contain off-topic changes (use other PRs for other changes).
- [ ] Applied any dependency changes to both
setup.cfg
(andconda-environment.yml
if present). - [ ] Tests are included (or explain why tests are not needed).
- [ ]
CHANGES.md
entry included if this is a change that can affect users - [ ] Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
- [ ] If this is a bug fix, PR should be raised against the relevant
?.?.x
branch.