metaflow
metaflow copied to clipboard
Netflix
Addition of conditional branching of Tasks
Pull request for adding the functionality of branching of tasks based on a condition.
The following changes are made -
- With the already existing implementation, a join operation is checked if there is an extra parameter passed to the function. But a join task for split-or takes only one input, the input from the branch being run. Hence another function traverse_fix_split_or is run after split_or to fix the graph.
- Conditions are added for scheduling of out tasks of split-or and the scheduling of split-or join.
@thesillystudent Thanks for the PR. Conditional branching is kind of a mine-field in the sense that it makes reasoning about workflows (and state transitions) non-trivial for users. Do you have specific instances of workflows that necessarily require conditional branching?
@savingoyal I have a few use cases which I'm currently using. Let's take the case of model invocation. Suppose if the result of model-1 is +ve then I may want to run some more steps, like maybe invoking some other models. But if the result of model-1 is -ve I want to move to the end.
@savingoyal I have a few use cases which I'm currently using. Let's take the case of model invocation. Suppose if the result of model-1 is +ve then I may want to run some more steps, like maybe invoking some other models. But if the result of model-1 is -ve I want to move to the end.
Hi @thesillystudent Do you have any information about merging this PR? I'm creating a Data pipeline where I need conditional branching, it will be good to have this feature in the nearest future.
@hovhanns I don’t have any plans as such. I can complete it over some weekends if enough people ask for it and I get a goahead from the devs
@thesillystudent I am converting this PR to draft since there is no immediate need for this functionality. Please reach out to us at slack.outerbounds.co if you would like to work on this PR.