amoro
amoro copied to clipboard
[AMORO-3943] Shade all third party classes for spark runtime
Why are the changes needed?
Close #3943.
all not shaded class
jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|wc -l
1077
org.apache.thrift
jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep 'org/apache/thrift'|wc -l
179
org.apache.commons.io
jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep 'org/apache/commons/io' | wc -l
201
com.github.benmanes
jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep 'com/github/benmanes' | wc -l
692
org.threeten from orc-core https://github.com/apache/orc/tree/main/java/core/src/java/org/threeten
jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep 'org/threeten' | wc -l
4
others:
jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep -v 'org/apache/thrift' |grep -v 'org/apache/commons/io' |grep -v 'com/github/benmanes' |grep -v 'org/threeten'
META-INF/versions/9/module-info.class
Brief change log
- Shade all third party classes for spark runtime
How was this patch tested?
-
[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
-
[x] Add screenshots for manual tests if appropriate
./mvnw clean package -Pspark-3.5 -Pscala-2.12 -pl amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5 -am -DskipTests
all classes shaded
./mvnw clean package -Pspark-3.3 -Pscala-2.12 -pl amoro-format-mixed/amoro-mixed-spark/v3.3/amoro-mixed-spark-runtime-3.3 -am -DskipTests
- [x] Run test locally before making a pull request
Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
cc @zhoujinsong @xxubai
cc @zhoujinsong