Updating to Spark 3.2.0

[jordyperlee]

We are looking to update to Scala 2.13 (and eventually Scala 3), but that requires us to update Scala to 3.2.0. Is there any plan on updating the Spark base image for the operator?

[mbaron2]

Hello,

Any update of this subject ?

[afranzi]

We would be interested also in updating to 3.2.X

[amalkhed]

Hello, We would also like to use spark 3.2.. Any update on support for this version ?

[josecsotomorales]

Hey Team, any plans to release an operator version that officially supports 3.2? I checked that it actually works with a custom image of spark 3.2 and the latest operator image (3.1.1), but did not deeply tested.

[cchantep]

May we get an answer at some point?

[Felix-neko]

Hi folks! Can you please officially support Spark 3.2.0?

[leandro-rouberte]

+1

[sharonorman78]

Hi

any news regarding this issue?

many Q's here - can you estimate when will it be supported? thanks :)

[allenhaozi]

I built an image based on spark-3.2.1-bin-hadoop3.2 Is that a problem?

[mbaron2]

@allenhaozi your image works ? Can you share it ?

thanks ! :)

[allenhaozi]

@mbaron2

I am currently in the test environment and can work Because it's still in beta, I can email you if you want

The main reference is the following items:

• spark-docker-file
• pyspark-notebook-docker-file
• spark-operator-docker-file

[sergialonsaco]

Is there any new regarding this update to 3.2.0 officially?

[josecsotomorales]

Spark 3.3.0 was released, I have a custom image of 3.2.1 running with the operator without any issues, and will experiment with 3.3.0 this week.

[vburenin]

In most cases using future version of spark is smooth, the main difference is typically in some configuration parameters that are being passed to the spark driver which you can also set in the spec if necessary. So folks, just build your own image and give it a try.

[wolliq]

Hi @vburenin I built a Spark 3.2.1 image which works fine but in some cases the Spark application shows this trace

22/08/18 11:14:17 WARN WatcherWebSocketListener: Exec Failure java.io.EOFException at okio.RealBufferedSource.require(RealBufferedSource.java:61) at okio.RealBufferedSource.readByte(RealBufferedSource.java:74) at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117) at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101) at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 22/08/18 11:14:18 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.) io.fabric8.kubernetes.client.WatcherException: too old resource version: 12998019 (13034876)

and hangs for hours without failing.

Could this be somehow related to the image version? Thx

[rahulbhatia2702]

Spark 3.3.0 was released, I have a custom image of 3.2.1 running with the operator without any issues, and will experiment with 3.3.0 this week.

Hi @josecsotomorales can you share the dockerfile for this?

[sousaaguilherme]

I would also be interested in this!

[allenhaozi]

I built it myself for testing allenhaozi/base-pyspark-3.2.1-py-v3.8:v0.1.0

[vburenin]

if anybody curious, it all works just fine if your target execution image is spark 3.3.

[josecsotomorales]

I'm actually using the operator with spark 3.3 with a custom image, see this issue: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/1559