dbt-core
dbt-core copied to clipboard
[CT-997] [Feature] Support dbt.var + dbt.env_var in Python models
Is this your first time submitting a feature request?
- [X] I have read the expectations for open source contributors
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
Describe the feature
Copying from thread in #beta-feedback-python-models:
A quick thought, why not expose a
dbt.var
function (and similar for env var) rather than overloading config? If you apply the same limitations as for config function (ie. Only literal args) should be pretty straightforward to statically analyse which vars need to be injected?
It's true! The mechanism for dbt.var
and dbt.env_var
could be just the same as dbt.config.get
. We detect the literal arguments passed in:
https://github.com/dbt-labs/dbt-core/blob/eb72dbf32aa96394d599c28db439e0b2c6fcc135/core/dbt/parser/models.py#L113-L117
https://github.com/dbt-labs/dbt-core/blob/eb72dbf32aa96394d599c28db439e0b2c6fcc135/core/dbt/parser/models.py#L182-L184
And then provide those values at "compile" time (which we're currently doing via Jinja... gross), as new methods (var
+ env_var
) on the dbtObj
class:
https://github.com/dbt-labs/dbt-core/blob/eb72dbf32aa96394d599c28db439e0b2c6fcc135/core/dbt/include/global_project/macros/python_model/python.sql#L33-L40
https://github.com/dbt-labs/dbt-core/blob/eb72dbf32aa96394d599c28db439e0b2c6fcc135/core/dbt/include/global_project/macros/python_model/python.sql#L71-L77
Describe alternatives you've considered
Not doing this, and continuing to ask users (as we are currently) to pass those vars + env_vars in as configs in .yml
files:
version: 2
models:
- name: my_python_model
config:
materialized: table
target_name: "{{ target.name }}"
specific_var: "{{ var('SPECIFIC_VAR') }}"
specific_env_var: "{{ env_var('SPECIFIC_ENV_VAR') }}"
Who will this benefit?
Usability and ergonomics for people developing and configuring Python models, especially ones with more advanced conditional logic or env-specific behavior
Are you interested in contributing this feature?
No response
Anything else?
To be clear, the environment variable values would need to be the ones evaluated during dbt parsing. (They are NOT the environment variables set in the runtime environment, e.g. Snowpark.)
That's slightly different from how environment variables work in SQL models today, since the {{ env_var() }}
Jinja is re-rendered at compile/execute runtime.
But I think it probably should work this way across the board, where all inputs (including env vars) are resolved during parsing and pre-execution.
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.