Validate and add support for Spark 4.0.0

Open fdosani opened this issue 6 months ago • 1 comments

Spark 4.0.0 is out now. Should spend some time to understand any differences and test if we are compatible with the latest version: https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-35-to-40

Jun 12 '25 17:06 fdosani

Documenting some of the changes I'm seeing so far:

https://spark.apache.org/docs/4.0.0/sql-ref-ansi-compliance.html#cast (spark.sql.ansi.enabled: false needs to be set it seems)

Jun 20 '25 20:06 fdosani