dbt-utils icon indicating copy to clipboard operation
dbt-utils copied to clipboard

dbt run fails for microbatch strategy with dbt_utils.star()

Open johan-cruxdata opened this issue 6 months ago • 0 comments

Describe the bug

dbt run fails in Databricks when attempting incremental microbatch build of a model that utilises dbt_utils.star() or dbt_utils.get_filtered_columns_in_relation(), which in turn rely on adapter.get_columns_in_relation(). Not yet tested on other databases.

Steps to reproduce

Create a model that uses, for example, dbt_utils.star() and configure the model with

config(
    materialized='incremental',
    incremental_strategy='microbatch',
    event_time='dt',    
    begin='2025-06-01',
    batch_size='day'
)

Expected results

Successful run

Actual results

Databricks throws an error: [PARSE_SYNTAX_ERROR] Syntax error at or near '('.

Screenshots and log output

Pseudocode for SQL written against Databricks:

describe table (
  select
    *
  from
    (
      select
        *
      from
        `catalog`.`schema`.`table`
      where
        false
      limit
        0
    ) _dbt_limit_subq_stg_foo
  where
    dt >= '2025-06-16 00:00:00+00:00'
    and dt < '2025-06-17 00:00:00+00:00'
) _dbt_et_filter_subq_stg_foo

System information

The contents of your packages.yml file: dbt_utils

Which database are you using dbt with?

  • [ ] postgres
  • [ ] redshift
  • [ ] bigquery
  • [ ] snowflake
  • [x] other (specify: Databricks)

The output of dbt --version:

1.9.3

Additional context

Are you interested in contributing the fix?

Omitting the table from the describe statement results in a successful run, as does using a CTE to create a named table (i.e., with foo as ()) and then running describe table foo. Cross-database compatibility not tested, however.

johan-cruxdata avatar Jun 17 '25 10:06 johan-cruxdata