apache
Bump to Parquet 1.14.1
The tests are failing, but that's because Jackson ships with JDK21-specific code, and the shadow plugin cannot cope with that. There is a fork of the plugin, but that one requires Gradle 8.3+. Gradle plugins are blocked because revapi is not being maintained and does not work with Gradle >=8.1.1. @jbonofre is digging into bumping Gradle by running RevAPI standalone 🙏 I ran ./gradlew clean test locally and everything passed.
Yes, I will move forward on the Gradle/revapi PR (a bit busy with a few Avro stuff 😄 ).
Looks like there are erroneously issues with the duplicate classes, but they are JVM specific:
# Danger! Multiple jars contain identically named classes. This may cause different behaviour depending on classpath ordering.
# Run ./gradlew checkClassUniqueness --write-locks to update this file
## runtimeClasspath
[com.fasterxml.jackson.core:jackson-core, org.apache.parquet:parquet-jackson]
- META-INF.versions.11.com.fasterxml.jackson.core.io.doubleparser.BigSignificand
- META-INF.versions.11.com.fasterxml.jackson.core.io.doubleparser.FastDoubleSwar
- META-INF.versions.11.com.fasterxml.jackson.core.io.doubleparser.FastIntegerMath
- META-INF.versions.17.com.fasterxml.jackson.core.io.doubleparser.FastDoubleSwar
- META-INF.versions.17.com.fasterxml.jackson.core.io.doubleparser.FastIntegerMath
- META-INF.versions.21.com.fasterxml.jackson.core.io.doubleparser.FastDoubleSwar
- META-INF.versions.21.com.fasterxml.jackson.core.io.doubleparser.FastIntegerMath
Unfortunally, we cannot upgrade gradle-baseline-java because that would require Java11+
@Fokko do you want me to check if I can do a baseline gradle plugin version JDK8 compliant dealing with that duplicated classes ?
Feel free to dig into it. The current version that we use is from November 2021: https://github.com/palantir/gradle-baseline/releases/tag/4.42.0 :(
@Fokko ok let me take a look. I think it would be great to include Parquet update in Iceberg 1.6.0 release.
I'm also good to leave this one out for the 1.6.0 release. I think it is going to be quite a bit of work.
@Fokko do you want me to check if I can do a baseline gradle plugin version JDK8 compliant dealing with that duplicated classes ?
@jbonofre: I don't think we should maintain private forks of these.
Dropping JDK8 seems like a good idea, so we can bump the gradle-baseline-java which handles the multi-version-jar.
@ajantha-bhat my previous comment was not about doing a private plugin fork, it was more about fixing the plugin. Palantir seems to still maintain the baseline plugin, so I can provide a PR.
@findepi Looks like some false positive from the shadow plugin that's unable to coop with the Java-version specific files. Have to dig deeper into this.