dbt-databricks icon indicating copy to clipboard operation
dbt-databricks copied to clipboard

Liquid cluster columns are updated on every run, even when there is no change

Open krifra1234 opened this issue 1 year ago • 8 comments
trafficstars

Describe the bug

When using liquid clustering the cluster columns in the deltalake table are update every time the dbt model is ran, even if the cluster columns are not changed in the config.

Steps To Reproduce

Create a DBT model with the follwoing config: materialized= 'incremental', incremental_strategy= 'append', liquid_clustered_by= ['columnname']

Run the dbt model multiple times and look for the operation "CLUSTER BY" in the deltalake table history. Fine the column “Operation Parameters” and you will see something similar to this for every run:

{ "oldClusteringColumns": "columnname", "newClusteringColumns": "columnname" }

Expected behavior

I would expect the CLUSTER BY operation not to run when the cluster columns are not changed.

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

System information

The output of dbt --version:

(dbt_1.8.5) PS C:\repo\dbt-bitechno> dbt --version
Core:
  - installed: 1.8.5
  - latest:    1.8.6 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.8.5 - Update available!
  - spark:      1.8.0 - Up to date!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

The operating system you're using: Microsoft Windows 11 Enterpris

The output of python --version: Python 3.10.14

Additional context

Add any other context about the problem here.

krifra1234 avatar Sep 20 '24 10:09 krifra1234