airflow-toolkit
airflow-toolkit copied to clipboard
dbt Cloud & airflow: best of friends
Merge Request Description
- Demonstrate how dbt Cloud can work well with airflow
- Give people easy code examples to follow and copy for themselves
- Using XCOM push and pull with
KubernetesPodOperator
-
Airflow 2.1.0
for improved UI and speed
Checklist
- [x] I ran this code locally(attached screenshot)
- [x] I wrote the necessary PASSING tests(attached screenshot)(I performed a dry run)
- [x] I covered my code with type-hints
Passing dbt Cloud job run Checkov and Terragrunt CICD checks are intended to fail
Considerations
- I had to make a manual git clone https for my local setup which will break the Google Cloud setup inherently because Cloud Composer can't freely access the public internet with the current configs. I have to allow Google Cloud access to the dbt Labs github organization which is too risky for this project.
- I had to enable cloud source repos api manually, but this is a trivial issue given the above. What matters is that I get my airflow toolkit to work locally
- This will be a good demo to show off how dbt Core can work well with Google Cloud AND a side by side comparison of how airflow can work with dbt Cloud smoothly.
- ONLY
Toolkit #1
is in scope for the changes here - This pull request SHOULD NOT be merged until the cloud source mirror repo issue is resolved
-
Airflow 2.0
is MUCH FASTER and the UI is much cleaner and modern compared toAirflow 1.0