metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard

Implementation for triggering a SFN execution from the result of a previous SFN execution

Open iiAnderson opened this issue 3 years ago • 4 comments

An implementation to allow SFN Executions to be triggered from the finishing state of another SFN execution. I have been using this to trigger multiple consumer Flows when a specific Producer flow finishes. For example:

Hourly -> DataCleaningFlow DataCleaningFlow -> Analysis1 DataCleaningFlow -> Analysis2

This ensures that when Analysis1/2 run, they will be able to obtain the data from the DataCleaningFlow, no need to guess at when it might end.

To implement this, I've modified the @schedule annotation to take a flow_name. This could be extended to add multiple flows and wait for their state (currently only support SUCCEEDED from SFN).

Closes #837

iiAnderson avatar Nov 25 '21 17:11 iiAnderson

Syntax still requires formatting.

iiAnderson avatar Nov 25 '21 17:11 iiAnderson

@iiAnderson I am taking the next couple of weeks off. I can take a look at this once I am back. Would that work?

savingoyal avatar Nov 30 '21 04:11 savingoyal

No rush @savingoyal, just a feature I've found useful!

iiAnderson avatar Dec 02 '21 19:12 iiAnderson

Hi @savingoyal, any updates on this PR? This would be a really useful feature to have!

npow avatar Feb 22 '22 16:02 npow