[Feature] `env_var` does not work in an object under the `vars` section of `dbt_project.yml`

[joseph-chan-thoughtworks]

trafficstars

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

vars:
  STAGING:
    database: "{{ env_var('STAGING_DATABASE') }}"
    schema: "{{ env_var('STAGING_SCHEMA') }}"

This fails due to unexpected "{". However,

vars:
  STAGING_DATABASE: "{{ env_var('STAGING_DATABASE') }}"
  STAGING_SCHEMA: "{{ env_var('STAGING_SCHEMA') }}"
  STAGING:
    database: "{{ var('STAGING_DATABASE') }}"
    schema: "{{ var('STAGING_SCHEMA') }}"

Expected Behavior

env_var within an object should parse correctly as well.

Steps To Reproduce

1. add STAGING_DATABASE and STAGING_SCHEMA to your environment
2. create a dbt project and add the above code block to dbt_project.yml
3. run dbt run or a linter for example sql_fluff

Relevant log output

.exceptions.DatabaseException: Database Error
  001003 (42000): SQL compilation error:
  syntax error line 1 at position 15 unexpected '{'.
  syntax error line 1 at position 25 unexpected '('.
  syntax error line 1 at position 46 unexpected '}'.

Environment

- OS: MacOX 11.6
- Python: 3.8
- dbt: 0.20.x

What database are you using dbt with?

snowflake

Additional Context

No response

[jtcohen6]

@joseph-chan-thoughtworks Thanks for opening!

The values of vars, defined in dbt_project.yml, aren't rendered today. They have to be defined as static values. As such, dbt doesn't resolve the env_var() or var() calls in your definitions — it just stores the whole string value, curly braces and all.

I believe this is a duplicate of https://github.com/dbt-labs/dbt-core/issues/3105. We should update the docs here to make clear that vars don't support rendering.

Could I ask for a bit more context about the use case you're after? Why are you creating a var to package env var values, rather than just referencing the env vars themselves in your downstream configuration / models / etc?

I appreciate the ergonomics of being able to synthesize target or env_var values to assign vars, and avoid duplication of more verbose expressions elsewhere; but I don't think there's anything that's functionally impossible because of this limitation.

[jeremyyeo]

Hey @jtcohen6, one possible use case for this is that packages tend to use variables to control "features". Let's say we use the hubspot package in our project. The way one would disable models in that package is by setting vars that are relevant to that package:

# dbt_project.yml

vars:
  hubspot_marketing_enabled: false
  hubspot_contact_enabled: false
  hubspot_sales_enabled: true
  hubspot_company_enabled: true
...

So to override those many vars, you would have to have some super long string in your run command:

dbt run --vars {hubspot_marketing_enabled: true, hubspot_contact_enabled: true, hubspot_sales_enabled: false, hubspot_company_enabled: false, ...}

Which is less than ideal.

And as of now, the following wont work:

Standard variables

# dbt_project.yml

vars:
  hubspot_marketing_enabled: "{{ var('control_flag_a') }}"
  hubspot_contact_enabled: "{{ var('control_flag_a') }}"
  hubspot_sales_enabled: "{{ var('control_flag_b') }}"
  hubspot_company_enabled: "{{ var('control_flag_b') }}"
...

dbt run --vars '{control_flag_a: true, control_flag_b: false}'

Environment variables

# dbt_project.yml

vars:
  hubspot_marketing_enabled: "{{ env_var('control_flag_a') }}"
  hubspot_contact_enabled: "{{ env_var('control_flag_a') }}"
  hubspot_sales_enabled: "{{ env_var('control_flag_b') }}"
  hubspot_company_enabled: "{{ env_var('control_flag_b') }}"
...

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions[bot]]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.

[adam-wrobel]

I would greatly appreciate this feature to exist.

[franviera92]

I would also like this change, if I need to specify a vars per environment, how would I apply this change currently?

[timvw]

I would also like this change, if I need to specify a vars per environment, how would I apply this change currently?

That's why I avoid vars (I only use them for setting environment unaware/unrelated things)

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions[bot]]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.