open-api: Build runtime jar for test fixture

[ajantha-bhat]

Since the REST catalog TCK is merged (https://github.com/apache/iceberg/pull/10908), we can have a runtime jar from the test fixture which can be used for building the docker image of rest catalog adapter.

PR for docker image will be raised in the follow up (already working on it)

[ajantha-bhat]

@kevinjqliu: I wasn't sure because of https://github.com/apache/iceberg/pull/9871.

I can add it. If we agree that it is useful.

[kevinjqliu]

#9871 is superseded by #10908

See https://github.com/apache/iceberg/pull/10908/files#diff-d11c4195b39c7a98f6f1ed9ab99d1845ca19d297574e92b836c95ff6aa6e1701L25-L29

[kevinjqliu]

I initially thought this was creating the jar for the TCK, but it looks like its for the RESTCatalogServer class instead. I wonder if it would be easier to work with if the class is moved out of the iceberg-open-api project

[ajantha-bhat]

I initially thought this was creating the jar for the TCK, but it looks like its for the RESTCatalogServer class instead. I wonder if it would be easier to work with if the class is moved out of the iceberg-open-api project

I think the current design of keeping them as test fixture make sense to me. As it is not a production code.

[ajantha-bhat]

Retriggering build because of connection error

Downloading https://services.gradle.org/distributions/gradle-8.10.2-bin.zip

Error: Exception in thread "main" java.io.IOException: Server returned HTTP response code: 500 for URL: https://github.com/gradle/gradle-distributions/releases/download/v8.10.2/gradle-8.10.2-bin.zip at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2011) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1602) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223) at org.gradle.wrapper.Install.forceFetch(SourceFile:2) at org.gradle.wrapper.Install$1.call(SourceFile:8) at org.gradle.wrapper.GradleWrapperMain.main(SourceFile:67) Error: Process completed with exit code 1.

[ajantha-bhat]

Any more suggestions for this? PR looks to be ready.

[Fokko]

@ajantha-bhat First of all, thanks for working on this, and sorry for the late reply as I was on parental leave. Unfortunately, there are issues with this. I just ran the following to check what's being shipped:

./gradlew build -x test -x integrationTest

And noticed that we ship quite a bit in the fat jar:

ls -lah open-api/build/libs/iceberg-open-api-*
-rw-r--r--  1 fokko.driesprong  staff   6.0K Oct 28 21:12 open-api/build/libs/iceberg-open-api-1.7.0-SNAPSHOT-javadoc.jar
-rw-r--r--  1 fokko.driesprong  staff   6.0K Oct 28 21:12 open-api/build/libs/iceberg-open-api-1.7.0-SNAPSHOT-sources.jar
-rw-r--r--  1 fokko.driesprong  staff   9.5K Aug 27 20:18 open-api/build/libs/iceberg-open-api-1.7.0-SNAPSHOT-test-fixtures.jar
-rw-r--r--  1 fokko.driesprong  staff    10K Oct 28 21:12 open-api/build/libs/iceberg-open-api-1.7.0-SNAPSHOT-tests.jar
-rw-r--r--  1 fokko.driesprong  staff   109M Oct 28 21:12 open-api/build/libs/iceberg-open-api-test-fixtures-runtime-1.7.0-SNAPSHOT.jar

When looking into the runtime:

mkdir /tmp/vo
mv open-api/build/libs/iceberg-open-api-test-fixtures-runtime-1.7.0-SNAPSHOT.jar /tmp/vo/iceberg-open-api-test-fixtures-runtime-1.7.0-SNAPSHOT.jar
cd /tmp/vo
unzip iceberg-open-api-test-fixtures-runtime-1.7.0-SNAPSHOT.jar

And looking at the content, it looks like we ship a LOT, much more than mentioned in the LICENSE. Including at least one licenses that we cannot ship as it is part of the category X:

ls -lah | grep -i lgpl
-rw-r--r--   1 fokko.driesprong  wheel   7.5K Feb  1  1980 LGPL-3.0.txt

Also for the release we probably want to track down where this comes from.

[ajantha-bhat]

@bryanck,

• I have cleaned up the dependencies
• Used license-report plugin to generate the report (plugin included in this PR) ./gradlew :iceberg-open-api:generateLicenseReport and report in build/reports/dependency-license/index.html
• Used root notice and license.

Now I need your render tool to format and append the notice and license to existing file.

[Fokko]

Thanks for fixing that issue @ajantha-bhat, this looks much better:

ls -lah open-api/build/libs/iceberg-open-api-*   
-rw-r--r--  1 fokko.driesprong  staff   6.0K Oct 30 12:54 open-api/build/libs/iceberg-open-api-1.7.0-SNAPSHOT-javadoc.jar
-rw-r--r--  1 fokko.driesprong  staff   6.0K Oct 30 12:54 open-api/build/libs/iceberg-open-api-1.7.0-SNAPSHOT-sources.jar
-rw-r--r--  1 fokko.driesprong  staff    10K Oct 30 12:54 open-api/build/libs/iceberg-open-api-1.7.0-SNAPSHOT-tests.jar
-rw-r--r--  1 fokko.driesprong  staff    49M Oct 30 13:26 open-api/build/libs/iceberg-open-api-test-fixtures-runtime-1.7.0-SNAPSHOT.jar

I did some digging around and didn't find any red flags. I did notice that we ship parts of Hadoop that we probably need (Filesystem, Configuration, etc), and the 3.3.6 has a known vulnerability, so good to bump that: https://github.com/apache/iceberg/pull/11428/

[Fokko]

This looks good, let's move this forward, and wrap up anything pending in https://github.com/apache/iceberg/pull/11283

[ajantha-bhat]

@Fokko: Thanks for the review. We can merge this and I can rebase the docker PR.

[ajantha-bhat]

@danielcweeks, @Fokko, @bryanck: Please take a look at the PR again. Thanks.

[Fokko]

Thanks @ajantha-bhat This looks good, if there are dangling issues, we can wrap them up in https://github.com/apache/iceberg/pull/11283

Let's move this forward, thanks @kevinjqliu, @jbonofre, @bryanck and @danielcweeks for the review 🙌