dbt-external-tables
dbt-external-tables copied to clipboard
Errors on `--empty´ Flag run
Describe the bug
Several statements use the {{ source() }} macro to gather the name of the table/pipe.
When running our external tables/pipe on the CI, we use the --empty flag.
Here the source name gets overridden by sub-select using where false limit 0.
Hence all these statements raises compilation errors.
Example:
create or replace table (select * from db.schema.name where false limit 0)
The dbt documentation recommends to use .render() for such cases: https://docs.getdbt.com/reference/commands/build#the-render-method
To have a proper, automated deployment we execute the stage_external_sources() macro as a pre_hook of the downstream staging model.
Thats why we have just one command with the additional empty flag.
Steps to reproduce
- Create snowpipe via yaml definition
- Create a sql model on top
- Add a pre_hook executing the snowpipe
- Run the model adding the
--emptyflag
Expected results
Actual results
Screenshots and log output
System information
The contents of your packages.yml file:
Which database are you using dbt with?
- [ ] redshift
- [x] snowflake
- [ ] other (specify: ____________)
The output of dbt --version:
dbt Cloud latest
The operating system you're using:
The output of python --version:
Additional context
Hello, any possible prioritization of this topic ? 🙏
We are facing this issue using bigquery.
Still facing this issue on BQ.
We use the following work around for big query metadata columns :
- add a macro
{% macro system_metadata(system_metadata_column, val="''") %}
{% if flags.EMPTY %}
{{ val }}
{% else %}
{{ system_metadata_column }}
{% endif %}
{% endmacro %}
- we use it to surround metadata column :
{{ system_metadata('_FILE_NAME') }}or{{ system_metadata('_PARTITIONTIME', 'CURRENT_TIMESTAMP()') }}
Thank you for the workaround @juoll-data !
We also encountered this issue. Any plans to resolve it?