ci-cd-for-data-processing-workflow
ci-cd-for-data-processing-workflow copied to clipboard
Handle building common dependencies before pipelines
Many organizations have build common data infrastructure packages that are used by many pipelines (e.g. convenience wrappers for spark, beam, airflow to enforce consistency across the team/org). This repo should leverage a more sophisticated build tool like Bazel to manage builds of these dependencies in the appropriate order.