airflow-toolkit icon indicating copy to clipboard operation
airflow-toolkit copied to clipboard

dbt Cloud & airflow: best of friends

Open sungchun12 opened this issue 3 years ago • 0 comments

Merge Request Description

  • Demonstrate how dbt Cloud can work well with airflow
  • Give people easy code examples to follow and copy for themselves
  • Using XCOM push and pull with KubernetesPodOperator
  • Airflow 2.1.0 for improved UI and speed

Checklist

  • [x] I ran this code locally(attached screenshot)
  • [x] I wrote the necessary PASSING tests(attached screenshot)(I performed a dry run)
  • [x] I covered my code with type-hints image image image Passing dbt Cloud job run Checkov and Terragrunt CICD checks are intended to fail

Considerations

  • I had to make a manual git clone https for my local setup which will break the Google Cloud setup inherently because Cloud Composer can't freely access the public internet with the current configs. I have to allow Google Cloud access to the dbt Labs github organization which is too risky for this project.
  • I had to enable cloud source repos api manually, but this is a trivial issue given the above. What matters is that I get my airflow toolkit to work locally
  • This will be a good demo to show off how dbt Core can work well with Google Cloud AND a side by side comparison of how airflow can work with dbt Cloud smoothly.
  • ONLY Toolkit #1 is in scope for the changes here
  • This pull request SHOULD NOT be merged until the cloud source mirror repo issue is resolved
  • Airflow 2.0 is MUCH FASTER and the UI is much cleaner and modern compared to Airflow 1.0

sungchun12 avatar Jun 29 '21 17:06 sungchun12