[FEATURE] Add support for run time parameters while deploying workflows
Currently user can specify custom params as hardcoded value
wf = Workflow(
"my-workflow",
default_cluster=cluster,
common_task_parameters={'my_key': 'my_value'}
)
And there is no way to pass params at deployment time
poetry run brickflow projects deploy -e local
Cloud Information
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] Other
Describe the solution you'd like
User will pass the params (max 10) at deployment time
poetry run brickflow projects deploy -e local -p key1=value1 -p key2=value2
These params will be available as string in task parameters as brickflow_runtime_params.
For above example, brickflow_runtime_params='{'key1': 'value1', 'key2': 'value2'}'
Describe alternatives you've considered User can get it working using ENVIRONMENT variable, not very ideal.
Additional context Add any other context or screenshots about the feature request here.
Disccussed with @pariksheet and @stikkireddy - We will be going with BRICKFLOW_PROJECT_PARAMS and use cli to pass the params. The parameters would be available through ctx and a helper function.