dbt-bigquery icon indicating copy to clipboard operation
dbt-bigquery copied to clipboard

[Bug] Cluster order for Materialized Views is mixed up

Open dennishendricks opened this issue 1 year ago • 4 comments

Is this a new bug in dbt-bigquery?

  • [X] I believe this is a new bug in dbt-bigquery
  • [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

After setting clusters for a materialized view via cluster_by in the Config macro, the order of the clusters is not adopted in the appropriate order as with other table forms, but is mixed according to a pattern that is not apparent.

Expected Behavior

Cluster order is adopted 1:1.

Steps To Reproduce

  1. Create a model as "materialized_view"
  2. Set clusters via cluster_by parameter
  3. See mixed cluster order in target -> run folder or result in BigQuery console

Example model:

{{
    config(
        materialized="materialized_view",
        cluster_by=[ "portal", "cu_name", "rt_name", "ids_in_container"],
    )
}}

select * from {{ ref('orders') }}

Result in target folder:

create materialized view if not exists `xxx`
    cluster by rt_name, ids_in_container, portal, cu_name

Relevant log output

No response

Environment

- OS: MacOS Sonoma 14.0
- Python: 3.11.1
- dbt-core: 1.7.8
- dbt-bigquery: 1.7.5

Additional Context

No response

dennishendricks avatar Feb 22 '24 09:02 dennishendricks

Thanks for reporting this @dennishendricks !

I didn't confirm or deny, but I suspect this is involved: https://github.com/dbt-labs/dbt-bigquery/blob/5501cd34b12965e0380952bdbb65fd52f49b49f5/dbt/adapters/bigquery/relation_configs/_cluster.py#L23

Since the cluster key is being converted into an unordered set rather than an ordered list, then the keys aren't guaranteed to preserve the order by the time we reach here.

dbeatty10 avatar Feb 22 '24 15:02 dbeatty10

Hi @dbeatty10, I have created a PR that fixes this bug. Would you mind taking a look at it? Thanks in advance.

thmswt avatar Jun 19 '24 06:06 thmswt

I added the triage label to get this back on the Product backlog given the new PR.

mikealfare avatar Jul 08 '24 17:07 mikealfare

Hi @mikealfare, I just updated the PR to the latest changes from main. The PR is ready for review.

thmswt avatar Jul 16 '24 07:07 thmswt