sedona icon indicating copy to clipboard operation
sedona copied to clipboard

Databricks Spark Configuration in Documentation

Open SteveScott opened this issue 1 year ago • 0 comments

In the Databricks Spark documentation here: https://sedona.apache.org/1.3.1-incubating/setup/databricks/

It says recommends the following Spark configuration:

spark.sql.extensions org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator org.apache.sedona.core.serde.SedonaKryoRegistrator

However, after speaking with Carto support, they said that the only configuration necessary was this:

spark.sql.extensions com.carto.analytics.toolbox.sql.SparkExtension
spark.databricks.geo.st.enabled true

Granted this includes a Carto analytics toolbox, and I don't know what is in that, but does the spark.databricks.geo.st.enabled true handle the KryoSerializer and KryoRegistrator now? With this configuration listed above, the KryoSerializer, KryoRegistrator were not needed. SedonaSqlExtensions and SedonaVizExtensions were also not needed. I just wanted to flag this as something may have changed on the Databricks side obviating the need for these configurations, replacing them with spark.databricks.geo.st.enabled = true.

SteveScott avatar Feb 05 '25 01:02 SteveScott