cylc
pool: check prereqs on task spawn
When we spawn a task, we currently check the DB to retrieve previous submissions. This is required for:
- Flow merge detection.
- Task re-run prevention (within the same flow).
- Preventing the accidental reincarnation of removed tasks.
Whilst we do check for previous submissions, we do not presently check task prerequisites. In some situations, task prerequisites which are satisfied in the DB, are left unsatisfied in the task pool when tasks are spawned. This can happen when tasks are added to the pool via means other than natural task satisfaction (e.g. https://github.com/cylc/cylc-flow/issues/5952).
Easy way to replicate this:
- Empty the task pool.
- Put the tasks back.
- Run the workflow on.
Sadly, I don't this it is sufficient to check the DB only when such interventions are performed, because tasks downstream of the ones being added which are yet to be spawned may also be missing this state and end up with (erroneously) partially satisfied prerequisites.
This is a bad bug as it makes it look to the user like tasks which have run, haven't. It's hard to explain, especially as multiple Cylc interfaces will provide erroneous information. Moreover it's very hard to recover from as the consequences may last as long as the longest inter-cycle dependency in the workflow.
There are efficiency concerns over requesting task prerequisites, however, we haven't ascertained that this would be a problem as yet, however, DB processing is definitely a bottleneck. One way to reduce these overheads might be to only request satisfied prerequisites from the DB. Merging this request in with the existing DB request would also improve performance. As would batching the requests where multiple tasks are spawned in the same main loop iteration.
This issue has relevance to workflow extension use cases (#5952), graph changes either by reload or restart
Sadly, I don't this it is sufficient to check the DB only when such interventions are performed, because tasks downstream of the ones being added which are yet to be spawned may also be missing this state and end up with (erroneously) partially satisfied prerequisites
Good point. It would be sufficient to check the DB only when the workflow stalls, but with significant cons:
- there could be a series of stalls until full recovery
- there could be a confusing period before the stall where tasks appear to be unsatisfied
Yes, this would be highly confusing.
I think the only way around this without checking prereqs for each spawn (not that I'm suggesting it per-se) would be to load in the task history from the DB and spawn on all outputs. If tasks already ran, nothing would happen, if they haven't then they will be spawned allowing the workflow to continue without the risk of artificial stall.
However, this would have the caveat that we would be spawning according to the current graph which may differ from the graph historical tasks were running according to (i.e. it could result in spawning errors).
This has now been discussed in a couple of different places, so removing the question label.
I think we have arrived at an agreement (e.g. see https://github.com/cylc/cylc-flow/issues/5952#issuecomment-2168109243) that DB prereqs should be checked on task spawn.
This will resolve a number of issues where in memory dependency brokering is insufficient to ensure consistency under the SoD approach. Such bugs can now be marked as superseded by this issue.
@oliver-sanders - Can you have a look at whether this test is a good starting point?
https://github.com/wxtim/cylc/pull/new/6143.check_prereqs_on_forced_spawn
This issue is similar to https://github.com/cylc/cylc-flow/issues/6822 which also requires fields to be loaded from the DB at spawn time. Suggest attempting the two together.
Another issue has been raised where this can happen as the result of a reload - https://github.com/cylc/cylc-flow/issues/7156
Checking the prereqs on task spawn might not be enough here (unless it is performed on reload?) as the task has already spawned.