Running dbt incremental model with multiple unique keys and merge strategy caused "collect" action failed error

[machov]

Describe the bug

Running dbt incremental model with multiple unique keys and merge strategy caused this error:

The "collect" action failed. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.

It's strange as it used to run fine in the same environment.

Steps To Reproduce

incremental model config:

{{
    config(
        materialized="incremental",
        unique_key=['current_date','name_col','id'],
        incremental_strategy="merge",
    )
}}

Expected behavior

The model should run smooth. This issue does not occur in databricks notebooks.

Screenshots and log output

Log/logs.dbt

merge into su_manrique_vargas_quesada.admin_analytics as DBT_INTERNAL_DEST
      using admin_analytics__dbt_tmp as DBT_INTERNAL_SOURCE
      on 
                  DBT_INTERNAL_SOURCE.current_date = DBT_INTERNAL_DEST.current_date
               and 
                  DBT_INTERNAL_SOURCE.name_col = DBT_INTERNAL_DEST.name_col
               and 
                  DBT_INTERNAL_SOURCE.id = DBT_INTERNAL_DEST.id  
      when matched then update set
         * 
      when not matched then insert *

System information

The output of dbt --version:

Core:
  - installed: 1.4.5
  - latest:    1.4.5 - Up to date!

Plugins:
  - databricks: 1.4.2 - Up to date!
  - spark:      1.4.1 - Up to date!

Also occurred with databricks 1.2.0

Core:
  - installed: 1.2.0
  - latest:    1.2.0 - Up to date!

Plugins:
  - databricks: 1.2.0 - Up to date!
  - spark:      1.2.0 - Up to date!

The operating system you're using: MacOS

The output of python --version: Python 3.8.13

[benc-db]

@mv1742 does this still repro on 1.6.x?

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.