amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[AMORO-3943] Shade all third party classes for spark runtime

Open turboFei opened this issue 1 month ago • 2 comments

Why are the changes needed?

Close #3943.

all not shaded class

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|wc -l                      
    1077

org.apache.thrift

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep  'org/apache/thrift'|wc -l
     179

org.apache.commons.io

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep  'org/apache/commons/io' | wc -l
     201

com.github.benmanes

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep  'com/github/benmanes' | wc -l
     692

org.threeten from orc-core https://github.com/apache/orc/tree/main/java/core/src/java/org/threeten

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep  'org/threeten' | wc -l      
       4

others:

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep -v 'org/apache/thrift' |grep -v 'org/apache/commons/io' |grep -v 'com/github/benmanes' |grep -v 'org/threeten'

META-INF/versions/9/module-info.class

Brief change log

  • Shade all third party classes for spark runtime

How was this patch tested?

  • [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • [x] Add screenshots for manual tests if appropriate

./mvnw clean package -Pspark-3.5 -Pscala-2.12 -pl amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5 -am -DskipTests     

all classes shaded image

./mvnw clean package -Pspark-3.3 -Pscala-2.12 -pl amoro-format-mixed/amoro-mixed-spark/v3.3/amoro-mixed-spark-runtime-3.3 -am -DskipTests     
image
  • [x] Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

turboFei avatar Nov 15 '25 02:11 turboFei

cc @zhoujinsong @xxubai

turboFei avatar Nov 19 '25 06:11 turboFei

cc @zhoujinsong

turboFei avatar Nov 25 '25 07:11 turboFei