dbt-clickhouse icon indicating copy to clipboard operation
dbt-clickhouse copied to clipboard

Incremental update errors with `NUMBER_OF_COLUMNS_DOESNT_MATCH` with tuple unique_key

Open tema-popov opened this issue 2 years ago • 1 comments

Hi, I can't create incremental table with tuple unique_key. I prepared two reproducible examples:

{{ config(order_by='(id1, id2)', engine='MergeTree()', materialized='incremental', unique_key='(id1, id2)') }}

select 1 as id1, 2 as id2

{{ config(order_by='(id1, id2)', engine='MergeTree()', materialized='incremental', unique_key='id1, id2') }}

select 1 as id1, 2 as id2

The error I get is

Code: 20. DB::Exception: Number of columns in section IN doesn't match. 2 at left, 1 at right. (NUMBER_OF_COLUMNS_DOESNT_MATCH) (version 22.1.3.7 (official build))

The reason seems to be excess brackets in generated query here:

where (id1, id2) not in (
    select (id1, id2) -- '(' and ')' are breaking query here
    from example_without_brackets__dbt_tmp
  )

Also, after error tables *__dbt_old, *__dbt_tmp don't clean up from DB and next time I'm trying to --run I get error Cache inconsistency detected: in rename, new key _ReferenceKey(database=None, schema='default', identifier='example_with_brackets__dbt_old') already in cache:

dbt.log clickhouse-version: 22.1.3.7

dbt --version

Core:
  - installed: 1.1.1
  - latest:    1.1.1 - Up to date!

Plugins:
  - clickhouse: 1.1.7 - Up to date!

tema-popov avatar Jul 21 '22 12:07 tema-popov

Current workaround -- instead of using tuple set unique_key as single expression, i. e. concat(toString(id1), '_', toString(id2))

tema-popov avatar Jul 21 '22 12:07 tema-popov