Co-ordinating Multiple Flows

[nabeelvalley]

Is there a way to coordinate multiple flows, or for a flow to state that it requires another flow to have run prior?

For example if I have a Data Preprocessing flow that I'd want to share data with to other flows/using the Client API it could be useful for a flow to state that it is reliant on another flow. Possibly via a decoration on the flow or something like that

And possibly also being able to kick off another flow in a declarative fashion

[savingoyal]

@nabeelvalley You can trigger workflow executions based on other workflow executions if you are using AWS Step Functions by setting custom rules on Amazon Event Bridge. We will soon provide a parameter TriggerOnFinish that will automate this setup and trigger a downstream workflow on AWS Step Functions on successful completion.

[nabeelvalley]

@savingoyal thanks for the information

is it possible to do this locally? or any plans to work on something like that?

[savingoyal]

@nabeelvalley We recently released a new feature that paves the way for introducing namespaced triggers - https://docs.metaflow.org/going-to-production-with-metaflow/coordinating-larger-metaflow-projects. I will follow up with a memo on event based triggers shortly.

[JulesBelveze]

Hi @nabeelvalley @savingoyal, I'd also like to trigger a flow B from a flow A, is there any update on this? :)

Cheers

[nabsabraham]

hi all -- any update on this?

[sphomme]

Hello everyone, Any update? Would be a great feature!

[savingoyal]

Yes indeed - we are actively working on - https://github.com/Netflix/metaflow/pull/1139

[hallsop]

@savingoyal #1139 appears to be closed, is it still being worked on? Cheers

[tuulos]

you can now use @trigger_on_finish (and other external events) to coordinate multiple flows. See here: https://docs.metaflow.org/production/event-triggering