kedro
kedro copied to clipboard
Add resume suggestion to parallel runner
Description
See #1795 for the original discussion and context surrounding this issue.
Currently, when Kedro is run using SequentialRunner
and a Node
creates an exception, Kedro will suggest resuming the run from the nearest Node
s with persisted input. This saves the user a great deal of time that would otherwise be wasted by running earlier Node
s whose output was successfully saved. Currently, no suggestion is made in the same case when Kedro is run using ParallelRunner
.
This is because, when a Node
reaches an exception while using ParallelRunner
, other nodes may still be in the process of running. The 'end state' of Node
s which have finished running when the exception is reached is not guaranteed due to the inherent stochasticity of the sequence with which Node
s are executed in a parallel scheme. Therefore, if 'resume-suggestion' logic were applied to ParallelRunner
, the suggestion would be inconsistent between runs, with the correct resume scenario suggestion only generated sporadically.
Context
This change will suggest a resume scenario to users who use ParallelRunner
, saving them a great deal of time that would otherwise be wasted in running Node
s unnecessarily.
Possible Implementation
Ensuring that all nodes that can be run are run before the 'end state' is reached and the exception is generated using (for example) joins.