ci-cd-for-data-processing-workflow icon indicating copy to clipboard operation
ci-cd-for-data-processing-workflow copied to clipboard

Handle building common dependencies before pipelines

Open jaketf opened this issue 5 years ago • 0 comments

Many organizations have build common data infrastructure packages that are used by many pipelines (e.g. convenience wrappers for spark, beam, airflow to enforce consistency across the team/org). This repo should leverage a more sophisticated build tool like Bazel to manage builds of these dependencies in the appropriate order.

jaketf avatar Jun 24 '20 16:06 jaketf