Select SQL endpoint at runtime via model configuration

[mcannamela]

trafficstars

Describe the feature

Databricks SQL has endpoints that are t-shirt sized, similar to Snowflake warehouses. Models with a lot of rows need larger endpoints, but these would be overkill for smaller models. When using the Snowflake adapter, it is easy to right-size the warehouse via configuration; however in the Databricks adapter, the endpoint is selected in the profile.

This feature would bring the Databricks adapter to parity with Snowflake, allowing the endpoint to be set via configuration and override what is in the profile.

Describe alternatives you've considered

The workaround is to use env vars in the profile and make multiple invocations of dbt. Large models can be tagged as such and selected via normal model selection mechanisms. But this could get quite sticky depending on the topology of the DAG, and it may not be possible to know a priori how many invocations you would need to cover the whole DAG.

Who will this benefit?

This will benefit anybody running Databricks SQL who has sufficient number and diversity of models such that some are much larger than others and would be more efficiently run on a larger endpoint.

Are you interested in contributing this feature?

Yes, I would likely just need some advice on approach to get started and occasional help if I get stuck.

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.

[amychen1776]

@mcannamela We are going to release this feature soon - would you be interested in beta testing?

[benc-db]

Available starting in 1.7.2