dbt-external-tables icon indicating copy to clipboard operation
dbt-external-tables copied to clipboard

Errors on `--empty´ Flag run

Open marvingeerken opened this issue 10 months ago • 6 comments

Describe the bug

Several statements use the {{ source() }} macro to gather the name of the table/pipe. When running our external tables/pipe on the CI, we use the --empty flag. Here the source name gets overridden by sub-select using where false limit 0. Hence all these statements raises compilation errors.

Example: create or replace table (select * from db.schema.name where false limit 0)

The dbt documentation recommends to use .render() for such cases: https://docs.getdbt.com/reference/commands/build#the-render-method

To have a proper, automated deployment we execute the stage_external_sources() macro as a pre_hook of the downstream staging model. Thats why we have just one command with the additional empty flag.

Steps to reproduce

  1. Create snowpipe via yaml definition
  2. Create a sql model on top
  3. Add a pre_hook executing the snowpipe
  4. Run the model adding the --empty flag

Expected results

Actual results

Screenshots and log output

System information

The contents of your packages.yml file:

Which database are you using dbt with?

  • [ ] redshift
  • [x] snowflake
  • [ ] other (specify: ____________)

The output of dbt --version: dbt Cloud latest

The operating system you're using:

The output of python --version:

Additional context

marvingeerken avatar Jan 23 '25 12:01 marvingeerken

Hello, any possible prioritization of this topic ? 🙏

kev-datams avatar Mar 05 '25 09:03 kev-datams

We are facing this issue using bigquery.

juoll-data avatar Mar 12 '25 08:03 juoll-data

Still facing this issue on BQ.

tlangton3 avatar Aug 13 '25 10:08 tlangton3

We use the following work around for big query metadata columns :

  • add a macro
{% macro system_metadata(system_metadata_column, val="''") %}
  {% if flags.EMPTY %}
    {{ val }}
  {% else %}
    {{ system_metadata_column }}
  {% endif %}
{% endmacro %}
  • we use it to surround metadata column : {{ system_metadata('_FILE_NAME') }} or {{ system_metadata('_PARTITIONTIME', 'CURRENT_TIMESTAMP()') }}

juoll-data avatar Aug 13 '25 10:08 juoll-data

Thank you for the workaround @juoll-data !

tlangton3 avatar Aug 13 '25 11:08 tlangton3

We also encountered this issue. Any plans to resolve it?

piotrdanielczyk-arrive avatar Nov 04 '25 11:11 piotrdanielczyk-arrive