spark-jobserver icon indicating copy to clipboard operation
spark-jobserver copied to clipboard

Published 20 hours ago • verified publisher icon spark-jobserver

Spark3

Open Ennosigaeon opened this issue 4 years ago • 6 comments

Pull Request checklist

  • [ ] The commit(s) message(s) follows the contribution guidelines ?
  • [ ] Tests for the changes have been added (for bug fixes / features) ?
  • [ ] Docs have been added / updated (for bug fixes / features) ?

Current behavior : #1269

New behavior : Spark Jobserver supports Spark 2.4, Spark 3.0 and Spark 3.1.

BREAKING CHANGES In Spark3, the option to execute multiple Spark contexts in a single JVM is not supported anymore (see apache/spark#23311). As this method was not recommended for a long time (see https://issues.apache.org/jira/browse/SPARK-2243), I removed this option also for Spark 2.4. The InProcessContextSupervisor (previously LocalContextSupervisor) still exists but should only be used for testing and local development and not in a production environment.

Other information:

  • Testing all three supported Spark version is a bit tricky. I used (sdkman)[https://sdkman.io/] to install all three versions locally. Next, I have created a Python 3.7 virtual env (also tested Python 3.8 for Spark 3.x) and install pyhocon in it. Before starting sbt, I have activated the venv and selected the appropriate Spark version with sdkman. Afterwards, you can use sbt as always.
  • By default Spark Jobserver is build against Spark 3.0.
  • Most of the changes are caused by removing Joda Time. Maybe it is easier to take a detailed look at the last two commits to see the "interessting" changes.
  • I have updated all outdated libraries that do not have breaking changes and replace joda time with java time.

This change is Reviewable

Ennosigaeon avatar Apr 08 '21 14:04 Ennosigaeon

Is there any news on this pull request? I don't know enough about the Travis CI to know what's failing or why, but is it related to #1363 ? We are looking forward to having the Spark 3.x support as our application depends heavily on SJS and that's the only holding us back from Scala 3. (Not that we need this tomorrow, but I'd love to update our head of engineering on the status.)

murraytodd avatar May 24 '21 17:05 murraytodd

@murraytodd There are some failing test cases that I sill have to fix (similar cause as #1363). Currently, I have no time to fix those.

Ennosigaeon avatar Jun 02 '21 08:06 Ennosigaeon

@murraytodd There are some failing test cases that I sill have to fix (similar cause as #1363). Currently, I have no time to fix those.

Thanks, @Ennosigaeon. That helps just to understand the status. (We're working on our roadmap planning, and it's great to see that the jump to Scala 3 is just around the corner.)

murraytodd avatar Jun 02 '21 15:06 murraytodd

@Ennosigaeon, do you know if there's any likelihood to get the flaky unit test issue resolved in the near future? Spark 3 support is starting to become an issue...

murraytodd avatar Feb 16 '22 14:02 murraytodd

We just faced the need to upgrade to Spark 3. Therefore, I have taken the time to update the MR to add support for Spark 3.4.

Ennosigaeon avatar Nov 07 '23 13:11 Ennosigaeon