[FEATURE] Option to deploy single workflow per project
Is your feature request related to a problem? Please describe.
Currently, when deploying a given project in Brickflow, it bundles and deploys all workflows under the project. Ideally, for versioned individual workflows we'd like to have the option to deploy just 1 workflow at a time. Ex.
Projects:
...Project1
......workflow_a.py
......workflow_b.py
...Project2
......workflow_a.py
......workflow_b.py
brickflow projects deploy --project Project1 -e local deploys workflow_a and workflow_b, a nice addition would be to have the option to only deploy workflow_a or workflow_b.
Describe the solution you'd like
brickflow projects deploy --project Project1 --workflow workflow_a -e local with --workflow as optional
Where the bundled path could then possibly include the workflow name
...user/project/workflow/env
Describe alternatives you've considered The alternative is creating a new project per workflow. In our case where we want a ETL pipeline's stages very carefully bundled, versioned, and released this yields 3 new projects at minimum per pipeline (with 3+ pipelines this yields 9+ minimum total projects increasing codebase size & complexity).
This is indeed very handy feature. However, we should enable this feature only for local development.
for higher environments, it does not make sense to deploy only set of workflows and should be part of version control.