[Bug] Incremental model run `CREATE` statement though the table exists

[yu-iskw]

Is this a new bug in dbt-bigquery?

• [X] I believe this is a new bug in dbt-bigquery
• [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I encountered an issue with the incremental model using dbt-bigquery. Upon investigating the executed BigQuery jobs, I observed unexpected behavior. Specifically, a CREATE OR REPLACE TABLE statement was executed with a WHERE clause using is_incremental(), despite the table’s existence. The expected behavior was the execution of a MERGE statement. This issue did not manifest across all incremental models; it was limited to a few.

Our scheduled jobs are managed through dbt on Google Cloud Composer, a fully managed Apache Airflow service. We leverage the KubernetesPodOperator to execute dbt commands within containers. Attempts to replicate the issue locally resulted in the expected execution of a MERGE statement, leaving the root cause of the discrepancy unclear.

The problem first appeared on 2024-02-20. To my knowledge, there were no changes or releases in dbt-core and dbt-bigquery around that time that could have contributed to this issue.

The following configurations define our incremental strategy.

{{ config(
    materialized="incremental",
    incremental_strategy='insert_overwrite',
    on_schema_change="append_new_columns",
    ...
   )
}}

SELECT ...
FROM ...
{% if is_incremental() %}
WHERE ...
{% endif %}

Expected Behavior

The expected behavior is to run a MERGE statement, as the table exists.

Steps To Reproduce

I haven't understand the steps to reproduce. We saw the issue only on scheduled jobs.

Relevant log output

No response

Environment

- Python: 3.11.4
- dbt-core: 1.7.8
- dbt-bigquery: 1.7.6

Additional Context

No response

[colin-rogers-dbt]

Thanks for surfacing, approximately how many times have you observed this behavior?

[colin-rogers-dbt]

Also, can you share the complete model config?

I can't reproduce locally and haven't heard this from other users, without more info I don't think there's much more we can do here. If this was transient it's possible this was some sort of network or other runtime issue. Will take a look at the logging PR you opened as that could help identify these in the future.