spark icon indicating copy to clipboard operation
spark copied to clipboard

Published 20 hours ago • verified publisher icon dotnet

Support for spark v3.3.1

Open natilivni opened this issue 2 years ago • 3 comments

Hello, can we expect support for spark v3.3.1?

natilivni avatar Nov 16 '22 07:11 natilivni

Dec 14 2022: Azure Synapse Runtime for Apache Spark 3.3 is now in Public Preview

  • Apache Spark to version 3.3.1
  • Delta Lake to version 2.1.0

Delta Lake version 2.1.0 Support for Apache Spark 3.3.

GeorgeS2019 avatar Feb 09 '23 13:02 GeorgeS2019

Hey,

We also require this package to support 3.3. We are using it on Cloudera CDH-7.1.8, which has Spark version 3.3.2. We are able to run our apps using Spark 2, but when trying to migrate to Spark3 we get this error: User class threw exception: java.lang.IllegalArgumentException: Unsupported spark version used: '3.3.2.3.3.7180.12-5'. Normalized spark version used: '3.3.2'. Supported spark major.minor version: '3.2'.

Thanks a lot!

dcosta91 avatar Jun 23 '23 12:06 dcosta91

@dcosta91 https://github.com/dotnet/spark/pull/1150

GeorgeS2019 avatar Jun 26 '23 21:06 GeorgeS2019