Netflix
Running flows from within flows
Are there any fundamental limitations inherent to executing a Flow as a step in a parent Flow? For example:
> cat nested_flow_child.py
from metaflow import FlowSpec, step
class ChildFlow(FlowSpec):
"""A simple child flow that processes some data."""
@step
def start(self):
print("Child flow: Processing data...")
self.result = "processed_data"
self.next(self.end)
@step
def end(self):
print(f"Child flow complete. Result: {self.result}")
if __name__ == "__main__":
ChildFlow()
> cat nested_flow_parent.py
from metaflow import FlowSpec, step, Runner
from nested_flow_child import ChildFlow
class ParentFlow(FlowSpec):
"""A parent flow that runs the ChildFlow using Runner."""
@step
def start(self):
print("Parent flow: Starting...")
self.next(self.run_child_flow)
@step
def run_child_flow(self):
print("Parent flow: Running child flow...")
# Use Runner to execute the child flow
child_flow_file = __file__.replace('_parent', '_child')
with Runner(child_flow_file).run() as running:
# Access the run object and get data from the end step
run = running.run
self.child_result = run['end'].task.data.result
print(f"Parent flow: Got result from child: {self.child_result}")
self.next(self.end)
@step
def end(self):
print("Parent flow: Complete!")
if __name__ == "__main__":
ParentFlow()
This ParentFlow has a single step that runs ChildFlow with Runner, and it appears to execute correctly.
Will this usage likely cause conflicts or problems with many other features though (params, configs, etc.)?
I'm aware of https://docs.metaflow.org/production/event-triggering/flow-events, and I have a good reason for wanting to use this pattern instead. Any advice would be much appreciated, thanks!
Yes, something like this is doable, and we use something similar at Netflix. One gotcha to keep in mind is that Metaflow copies over the env vars from the parent flow to the chile flow when using the runner.
Say, for instance the run_child_flow step has @pypi decorator, then it would basically set --environment=conda for the child flow as well.
Thank you @talsperre! When you say that you use something similar internally, does that mean there is a similar but better idiom worth considering? My goal is essentially to facilitate Flow composability, so it would be great to know if there are good ways to do that beyond @trigger_on_finish and nested flows like in my example above.
I am somewhat confused by the docs on Composing Flows since they're not actually about composing flows. They just describe ways to factor out shared functionality for reuse across flows. I certainly plan on doing that as well, but there are legitimate reasons to literally compose flows as well IMO. Please let me know if I'm missing something.
I would also love the same functionality. True flow composability is my biggest pain point with Metaflow as is.
Are there any plans to build out support for composability? Would love to find out more about what those plans are, or if there aren't any, discuss what it would take to get it on the roadmap and what the interface should look like.
In regards, to the presented pattern here, I suppose one issue is that the parent flow blocks, consuming resources, until the child flow completes. Or do I have that wrong?
We have work underway for offering more forms of composability. Here is a memo (details are somewhat outdated but the idea holds)
Thanks for sharing this! Glad to see there are concrete ideas. Would you be willing to share a rough timeline on when this might be available e.g. (Q2 2026)?
Hopefully much much sooner. We have a few other big features before this memo ships out.