airflow-dbt
airflow-dbt copied to clipboard
Abstract execution environment
As many out there might know every project is a whole different world. For this reason we find ourselves with a plethora of ways for running DBT. Though airflow-dbt
is an elegant solution we sometimes want to run this on a k8s pod, on cloud build or some other task execution environment. This is motivated by decoupling, security, different dbt versions (yes, dbt SEMVER is not consistent and minor versions break backwards compatibility so this is a thing), or simply centralising different orchestration tasks in a task-runner because it is already configured.
To accomplish this it would be interesting to:
- Abstract the command generation from the execution:
- Allow for dependency injection, passing the hook we want to use and run accordingly, pass a python function that receives the command (ala
GCSFileTransformOperator
) ... open for discussion
Class structure goes like this:
You can tell whichever Operator you choose to use which DbtHook do you want to use. Using the interface DbtBaseHook
ensures that they implement required methods