Compute Information in Target Information for Jinja templating

[RJaikanthEJ]

trafficstars

Hi,

I am trying to create a macro that defines different SQL warehouses for different models/environments. For ex - dev would almost always have 2x-small and production x-large. It can also be that different models require different warehouses depending on size.

I am looking for something similar to - https://github.com/dbt-labs/dbt-snowflake/issues/103#issuecomment-1250263420 but for the databricks adapter. I know we can set compute manually from https://docs.getdbt.com/reference/resource-configs/databricks-configs#selecting-compute-per-model but looking for a way to automate based on environment first and table size in the next iteration.

I could not find any documentation about what information can be accessed via the target variable. It would be helpful if you can point towards that.

[benc-db]

This is already possible without a macro. The compute definitions are per 'output' in your profile, so just use the same tag with a different definition that's matched to each env. ex:

my_profile:
  target: dev
  outputs:
    dev:
      type: databricks
      ...
      compute:
        small_tables:
          http_path: /sql/path/for/small_tables_in_dev

        big_tables:
           http_path: /sql/path/for/big_tables_in_dev

    prod:
      type: databricks
      ...
      compute:
        small_tables:
          http_path: /sql/path/for/small_tables_in_prod

        big_tables:
           http_path: /sql/path/for/big_tables_in_prod

[RJaikanthEJ]

Hi @benc-db

Thanks for the answer it is really helpful. Just for any future use case, is there a way to access the compute using a macro. Would we do it like target.compute.small_tables.http_path ?

[benc-db]

That's a good question that I don't know the answer to off the top of my head. I recommend experimenting with a SQL comment like:

-- {{ target.compute.small_tables.http_path }}

and see what shows up in the dbt.log. Also, someone on the dbt slack may know.