Databricks + DBT + Sedona compatibility

[alexeyegorov]

At our startup, we use the combination of Sedona and DBT on Databricks. However, the compatibility of latest Sedona versions with Databricks Runtime is not easy or clear to understand.

Following the documentation I though 1.7.2 and above should be compatible with DBR 16.4 LTS. I was able to run the Sedona spatial functions, but I could not the resulting dataset as Delta file. Using DBT I ran into a following error:

I have tried the following combinations:

• Sedona 1.7.2 + DBR 16.4 (above error)
• Sedona 1.8.0 + DBR 16.4 (above error)
• Sedona 1.8.0 + DBR 17.X (some Spark 4.0 incompatibility)

Does anyone here have experience with the setup? If there is something in development that we might test using our setup, I would be open to support that.

Until then, it seems that the Sedona 1.7.0 and DBR 15.4 is just the one stable solution.