dbt-databricks
dbt-databricks copied to clipboard
Liquid cluster columns are updated on every run, even when there is no change
Describe the bug
When using liquid clustering the cluster columns in the deltalake table are update every time the dbt model is ran, even if the cluster columns are not changed in the config.
Steps To Reproduce
Create a DBT model with the follwoing config: materialized= 'incremental', incremental_strategy= 'append', liquid_clustered_by= ['columnname']
Run the dbt model multiple times and look for the operation "CLUSTER BY" in the deltalake table history. Fine the column “Operation Parameters” and you will see something similar to this for every run:
{ "oldClusteringColumns": "columnname", "newClusteringColumns": "columnname" }
Expected behavior
I would expect the CLUSTER BY operation not to run when the cluster columns are not changed.
Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.
System information
The output of dbt --version:
(dbt_1.8.5) PS C:\repo\dbt-bitechno> dbt --version
Core:
- installed: 1.8.5
- latest: 1.8.6 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- databricks: 1.8.5 - Update available!
- spark: 1.8.0 - Up to date!
At least one plugin is out of date or incompatible with dbt-core.
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
The operating system you're using: Microsoft Windows 11 Enterpris
The output of python --version:
Python 3.10.14
Additional context
Add any other context about the problem here.