brickflow icon indicating copy to clipboard operation
brickflow copied to clipboard

[FEATURE] Add support for run time parameters while deploying workflows

Open pariksheet opened this issue 2 years ago • 1 comments

Currently user can specify custom params as hardcoded value

wf = Workflow(
    "my-workflow",
    default_cluster=cluster,
    common_task_parameters={'my_key': 'my_value'}
)

And there is no way to pass params at deployment time poetry run brickflow projects deploy -e local

Cloud Information

  • [ ] AWS
  • [ ] Azure
  • [ ] GCP
  • [ ] Other

Describe the solution you'd like User will pass the params (max 10) at deployment time poetry run brickflow projects deploy -e local -p key1=value1 -p key2=value2

These params will be available as string in task parameters as brickflow_runtime_params.

For above example, brickflow_runtime_params='{'key1': 'value1', 'key2': 'value2'}'

Describe alternatives you've considered User can get it working using ENVIRONMENT variable, not very ideal.

Additional context Add any other context or screenshots about the feature request here.

pariksheet avatar Nov 19 '23 18:11 pariksheet

Disccussed with @pariksheet and @stikkireddy - We will be going with BRICKFLOW_PROJECT_PARAMS and use cli to pass the params. The parameters would be available through ctx and a helper function.

asingamaneni avatar Feb 12 '24 16:02 asingamaneni