ci-cd-for-data-processing-workflow
ci-cd-for-data-processing-workflow copied to clipboard
jaketf
Cloud Build for Deploying Datapipelines with Composer, Dataflow and BigQuery
https://github.com/jaketf/ci-cd-for-data-processing-workflow/blob/87b938474d9edcc984a166ff140a3c9a996d3916/terraform/artifacts/main.tf#L55
The current example uses plugins for the contrived xcom compare operator. Instead this should be managed as a separate python module (e.g. in the DAGs folder on gcs ignored by...
https://godatadriven.com/blog/introducing-pylint-airflow/
We should support the use case where there are special CI only DAGs (e.g. those that run large scale integration tests) or prod only DAGs (e.g. one to destroy /...
The `composer/cloudbuild/bin/run_tests.sh` script should be able to set (or mock) all connection types. - [x] HTTP connection - [ ] Google Cloud Platform Connection - [ ] Google Cloud SQL...
## Use Case: Cloud Composer supports [private ip clusters](https://cloud.google.com/composer/docs/concepts/private-ip) which spins up a private IP GKE cluster in the customer project within a VPC network, many customers have org policies...
Many organizations have build common data infrastructure packages that are used by many pipelines (e.g. convenience wrappers for spark, beam, airflow to enforce consistency across the team/org). This repo should...