spark-excel
spark-excel copied to clipboard
crealytics
[BUG] Classpath problems with ThinFatJar
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
Using Spark-Excel 0.18.5 with Spark 3.3.1 in IntellJ, and get the following exception: java.lang.NoSuchMethodError: org.apache.logging.log4j.util.LoaderUtil.getClassLoaders()[Ljava/lang/ClassLoader;
The problems seems to be that Spark 3.3.1 has Log4j 2.17.2 as dependency, but Spark-Excel has shaded Log4J 2.19.0 into its jar (without renaming!). And there are many more dependencies shaded into it, even scala:
Shading in dependencies without renaming makes dependency management with Maven impossible. It potentially produces weird NoSuchMethodError and ClassNotFoundExceptions as conflicting versions are easily present in the classpath. Best practice is to let Maven solve dependencies, or rename shaded dependencies and remove them from the pom.
Why is it done like that for spark-excel?
Fixing this should make spark-excel much more compatible in my opinion.
Expected Behavior
- The default artifact published by spark-excel does not include any shaded dependency (or they are renamed). Needed maven dependencies are listed in the pom as currently done.
- An artifact including shaded dependencies can additionally be created and published using maven classifier. The classifier, e.g. "shaded", is then included in the published jar file, e.g. spark-excel_2.12-shaded-3.3.1_0.18.5.
Steps To Reproduce
No response
Environment
- Spark version: 3.3.1
- Spark-Excel version: 0.18.5
- OS: Windows
- Cluster environment: IntelliJ
Anything else?
No response
Hi @zzeekk, PRs to improve the packaging are very welcome! The current build uses Mill, so the build.sc file would be the appropriate place for any fixes.
Hi @nightscape, i finalized my tests on release 0.18.6-beta1, looks good to me. I have one more suggestion about the version number scheme, please see 0.18.6-beta1