Choose my compute when running DBT tests

[ybressler]

Describe the feature

When I run generic dbt commands (such as dbt debug), my project defaults to using the all_purpose_cluster defined in my connection profile.

When I execute dbt test --select a_big_model, my test utilizes the all_purpose_cluster which is too small for the computation – this causes my tests to fail because of resource issues. Is there a way for me to configure a specific cluster to be used when I run tests on a specific model?

Ideally, I can specify submission_method for a model's tests (think column level tests). Additionally, one-off tests could be executed with submission method as well. Alternatively, specifying http_path=... for a model's tests would be an appropriate solution too.

Related: SO Q -> https://stackoverflow.com/questions/76114948/dbt-databricks-specify-cluster-for-a-test

Are you interested in contributing this feature?

Sure. Probs best place to start is within the dbt test macro, no? I'll need some guidance as how to navigate, but would love to see this happen.

[benc-db]

As a workaround, you can just target a different profile with beefier compute when you run test. Otherwise, I think this is better raised with dbt-core.

[ybressler]

As a workaround, you can just target a different profile with beefier compute when you run test. Otherwise, I think this is better raised with dbt-core.

That's what I got in place. Though prevents me from running dbt build which is annoying.

[benc-db]

That's what I got in place. Though prevents me from running dbt build which is annoying.

Understood. We're working on supporting compute per model, and maybe that will help some, but specifying per dbt task is probably something dbt-core would need to fix.

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.