tispark
tispark copied to clipboard
Published
20 hours ago •
pingcap
pingcap
support Spark3.3
What problem does this PR solve?
support spark 3.3
What is changed and how it works?
- Add spark-3.3 profile
- Add Spark-3.3 module in spark-wrapper and add Code in spark-3.3
- Fix the code in spark-core to be compatible with spark3.0 3.1 3.2 3.3
- Change the pattern matching in
SetCatalogAndNamespacebecause the parament becomeResolvedNamespaceandResolvedDBObjectName. - Add method
dropNamespace(strings: Array[String])inTiCatalogfor spark3.3 change the parament ofdropNamespaceinSupportsNamespaces - Add method
parseQueryinTiParserfor spark3.3 addparserQueryinParserInterface.
- Change the pattern matching in
- Update
log4j-apiandlog4j-coreversion to 2.17.2 and exclude it in other dependencies. or you will get java.lang.NoSuchMethodError: org.apache.logging.log4j.core.appender.ConsoleAppender$Builder.setImmediateFlush in spark3.3 - Exclude slf4j-log4j12 dependency hadoop-client or you will find multiple bindings of
Log4jLoggerFactoryon the classpath - Update assembly to package spark-3.3 in spark wrapper
- Fix some tests
Check List
Tests
- IT
- /run-all-tests profile=spark-3.3.0
- /run-all-tests profile=spark-3.2.2
- /run-all-tests profile=spark-3.1.3
- /run-all-tests (spark 3.0.3)
- /run-all-tests profile=spark 3.3.0 tidb = v5.3.0 tikv = v5.3.0 pd = v5.3.0
- Use Test
- Package TiSpark with 3.0.3 and work with spark 3.0.3
- Package TiSpark with 3.1.3 and work with spark 3.1.3
- Package TiSpark with 3.2.2 and work with spark 3.2.2
- Package TiSpark with 3.3.0 and work with spark 3.3.0
[REVIEW NOTIFICATION]
This pull request has been approved by:
- shiyuhang0
- xuanyu66
To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.
The full list of commands accepted by this bot can be found here.
Reviewer can indicate their review by submitting an approval review. Reviewer can cancel approval by submitting a request changes review.
