Running dbt-databricks on a job cluster

[leo-schick]

trafficstars

Describe the feature

It is possible to run dbt SQL models inside a job cluster when:

• you use the dbt-spark adapter instead of dbt-databricks
• use the session profile configuration
• use Programmatic invocations to execute dbt

I would like to see:

• a configuration in the dbt-databricks adapter for session execution so I do not need to use dbt-spark
• an official way documented how to run dbt-databricks on a job cluster

Describe alternatives you've considered

I did several tests with the token based authentication but it looks like that in job clusters have another spark endpoint. Token based authentication does not work on a job cluster.

Something to note

Python models currently do not work with this approach.

Who will this benefit?

The Databricks license costs are reduced, because no general purpose cluster is necessary to run dbt inside databricks.