metaflow
metaflow copied to clipboard
Netflix
Allow specifying parameters when calling `step-functions create`
Currently step-functions trigger allows specifying parameters to invoke the step function, but not step-functions create (the default set of parameters is {}). This PR allows specifying parameters when calling step-functions create, similar to how step-functions trigger does it.
PS: I actually just copied the code from the trigger function.
I have one generic hourly flow, which is parameterized by different config files. I want to be able to specify the config file to use when setting up the event-bridge triggers, so that multiple configurations of the same flow can be triggered hourly.
I currently have a script that loops over all my config files, and calls python flow.py --branch $BRANCH step-functions create $CONFIG_FILE. I know about deploy time parameters, I don't really see how to use that unless I specify the config file via an environment variable. The way step-functions trigger does it seems more natural to me.
That makes sense. This PR would require a bit more thought if we also need to allow overriding of the parameters (set during step-functions create) through state-machine triggering as well.
Good callout. Is state-machine triggering being worked on? I see there's https://github.com/Netflix/metaflow/pull/840 but it seems to have stalled.
Here is the wider memo around event-triggering - https://docs.google.com/document/d/1liTvpACWKioCSQTUv5iO3g2AKuLu4x3EYFwEl43WAZU/edit.
Let me follow up on #840 with the changes needed to move it in the right direction. For this PR, let me play with this a bit - it's been a while since I dove into Step Functions, and want to ensure there isn't any other issue that will trip us. How about I follow up by mid-next week?
@npow circling back on this - the major issue with this feature is that parameters can potentially overlap with the CLI arguments of step-functions create like workflow-timeout, authorize etc. Given that we have an environment-based workaround available (deploy-time parameters), how about we continue with that for now?