airflow-dbt icon indicating copy to clipboard operation
airflow-dbt copied to clipboard

Abstract execution environment

Open dinigo opened this issue 3 years ago • 1 comments

As many out there might know every project is a whole different world. For this reason we find ourselves with a plethora of ways for running DBT. Though airflow-dbt is an elegant solution we sometimes want to run this on a k8s pod, on cloud build or some other task execution environment. This is motivated by decoupling, security, different dbt versions (yes, dbt SEMVER is not consistent and minor versions break backwards compatibility so this is a thing), or simply centralising different orchestration tasks in a task-runner because it is already configured.

To accomplish this it would be interesting to:

  1. Abstract the command generation from the execution:
  2. Allow for dependency injection, passing the hook we want to use and run accordingly, pass a python function that receives the command (ala GCSFileTransformOperator) ... open for discussion

dinigo avatar Oct 14 '21 09:10 dinigo

Class structure goes like this:

image

You can tell whichever Operator you choose to use which DbtHook do you want to use. Using the interface DbtBaseHook ensures that they implement required methods

dinigo avatar Oct 21 '21 11:10 dinigo