spark-operator icon indicating copy to clipboard operation
spark-operator copied to clipboard

When will operator support spark 3.3.0 or above?

Open jhonnli opened this issue 1 year ago • 7 comments

When will operator support spark 3.3.0 or above?

jhonnli avatar Oct 13 '23 02:10 jhonnli

it supports 3.4, not sure about 3.5

fei819 avatar Oct 17 '23 16:10 fei819

We are using it with Spark 3.3.2 as well

kori73 avatar Oct 19 '23 07:10 kori73

We are using it with Spark 3.3.2 as well

Which version are you using that supports v3.3.2?

jhonnli avatar Nov 17 '23 04:11 jhonnli

Which version are you using

Which version are you using

jhonnli avatar Nov 17 '23 04:11 jhonnli

Sorry, I missed it. I read the documentation carefully today. The latest releases support Spark 2.3 and up.

jhonnli avatar Nov 17 '23 06:11 jhonnli

So can we assume that

Spark 2.3 and up

includes all stable major versions of Spark, like 3.5? We already run it successfully with 3.3, but I'd appreciate it if the docs would give a bit more clarity on this. At least the version matrix prominently featured in the README.md implies that only 3.1.1 is supported.

daniel-mueller avatar Dec 15 '23 10:12 daniel-mueller

I'm running the Spark 3.2.1 with the latest operator but when i use the spark-avro 3.2.0, it said i'm running on the LEGACY spark and failed the job, do you have any idea? I'm running locally within Spark 3.2.1 with library is fine.

truongbk24 avatar Dec 27 '23 02:12 truongbk24

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Aug 14 '24 04:08 github-actions[bot]