spark-distcp icon indicating copy to clipboard operation
spark-distcp copied to clipboard

Adapt log4j2 to support Spark 3.3

Open pan3793 opened this issue 1 year ago • 1 comments

Why changes are needed?

Since Spark 3.3, it migrate from Log4J 1.x to Log4J 2.x, see details in SPARK-37814

Currently, the Logging has a hard dependency on Log4J1, which does not work on Spark 3.3 and above.

What are the changes?

This PR aims to use reflection to adapt both Log4J1 and Log4J2 to resolve such a compatible issue.

How was it verified?

A new test case TestChangeLoggerLevel is added, also updated the Azure pipeline definition to include the Spark 3.3

pan3793 avatar Dec 20 '23 12:12 pan3793

support bandwidth limit ?

melin avatar Mar 27 '24 12:03 melin