dbt-core
dbt-core copied to clipboard
[CT-998] [Bug] Provide a more helpful error when a project-level config is Undefined and cannot be deepcopied
Is this a new bug in dbt-core?
- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
DBT version 1.0.0 Plugins:
- bigquery: 1.0.0
I tried executing dbt run --select my_model
with the following syntax in my dbt_project.yml file:
dwh:
+database: db_name
dataset:
+schema: schema_name
+grant_access_to:
- project: "{{ 'dwh-prod' if target.name == 'prod' }}"
dataset: mydataset
The IF statement - project: "{{ 'dwh-prod' if target.name == 'prod' }}" is incomplete and will return the error message:
cannot pickle 'Undefined' object
Expected Behavior
If you want to grant access to this project you could just write: - project: prod. I expect the error message to be more descriptive than:
cannot pickle 'Undefined' object
If you get a hint on that the if-statement can't be parsed or something along these lines you could identify where the problem lies. I managed to reverse engineer this by finding https://github.com/dbt-labs/dbt-core/issues/2110 and concluding this was not caused by my models/refs and I started looking into the dbt_project.yml file.
Steps To Reproduce
- Be connected to a BigQuery db
- Replace the following targets in your project.yml files with ones relevant to your case: dwh: +database: db_name dataset: +schema: schema_name +grant_access_to: - project: "{{ 'dwh-prod' if target.name == 'prod' }}" dataset: mydataset
- Create a model that uses this database/schema combination
- Execute dbt run --select my_model and it should fail because of the incorrect IF-statement
Relevant log output
cannot pickle 'Undefined' object
### Environment
```markdown
- OS: macOS Monterey 12.4
- Python: 3.9.9
- dbt: 1.0.0
Which database adapter are you using with dbt?
bigquery
Additional Context
No response
@jesperdejby Thanks for opening, and for the clear restatement! I was able to reproduce this quite easily.
Indeed, the issue here is that, when target.name
is not prod
, the Jinja expression resolves to None
/Undefined
, which Python cannot deepcopy. (dbt needs to deepcopy project-level configs, during config resolution at parse time.)
A temporary fix is to provide an else
conditional value:
- project: "{{ 'dwh-prod' if target.name == 'prod' else 'something_else' }}"
I agree that the error message here is not very helpful in identifying the root cause. I'm going to reframe this issue as: dbt should provide a more helpful error when a project-level config is Undefined and cannot be deepcopied.
@jtcohen6 Thank you for the quick response and help 🙏 .
We're going to need to take a closer look to see if there is a reasonable place/way to determine when Undefined
is being returned from jinja before this can be estimated.
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.
Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.