spark-distcp
spark-distcp copied to clipboard
Adapt log4j2 to support Spark 3.3
Why changes are needed?
Since Spark 3.3, it migrate from Log4J 1.x to Log4J 2.x, see details in SPARK-37814
Currently, the Logging
has a hard dependency on Log4J1, which does not work on Spark 3.3 and above.
What are the changes?
This PR aims to use reflection to adapt both Log4J1 and Log4J2 to resolve such a compatible issue.
How was it verified?
A new test case TestChangeLoggerLevel
is added, also updated the Azure pipeline definition to include the Spark 3.3
support bandwidth limit ?