Config Value spark.sql.mapKeyDedupPolicy not Supported by Databricks SQL Warehouse

[ArtnerC]

Getting the error spark.sql.mapKeyDedupPolicy is not supported by Databricks SQL Warehouses when using ibis pyspark with a Databricks SQL Warehouse Cluster.

See: https://community.databricks.com/t5/data-engineering/spark-settings-in-sql-warehouse/td-p/7959

Set in do_connnect: https://github.com/ibis-project/ibis/blame/e425ad57899f8ebbea29b57bb53cedb40ebd7193/ibis/backends/pyspark/init.py#L180

self._session.conf.set("spark.sql.mapKeyDedupPolicy", "LAST_WIN")

Workaround could be as simple as:

try:
    spark.conf.set("spark.sql.mapKeyDedupPolicy", "LAST_WIN")
except Exception as e:
    if "not available" in str(e):
        print("Likely running in a SQL Warehouse")
    else:
        raise e  # Re-raise other exceptions

but I'm not sure what other approaches there might be.