Java 21 upgrade

[olivierboudet]

Hi,

Java 21 is GA, when can we hope to have an updated distroless image ?

thanks

[zapodot]

Looks like we need an image based on Debian 13 (trixie) to support this

[theFoxley]

Why ? There was java17 on debian11 even if it's not the default jre for this version

[loosebazooka]

If there's Java 21 on debian12, we'll add it. Perhaps a question to the debian maintainers?

[sansnom]

For me, it will be available. You can follow the package here: openjdk-21. But will need to wait for a few weeks (took one month for JDK 17). Minimum of 5 days between unstable and testing if I read well and then to stable.

EDIT: has been migrated to testing the 2023/09/26. Next step: testing->stable.

[fragaLY]

Hi,

Is it going to be released?)

[loosebazooka]

Is it available in debian?

[fragaLY]

Looks like we need an image based on Debian 13 (trixie) to support this

So we are going to wait till the 2025?

What is the problem to base on the debian12 (yes, I know support date of Debian 12 till 2028 and JDK21 till 2031)?

[dlorenc]

Shameless pointer, the wolfi-based Chainguard images are distroless as well and we just announced support for jdk 21:

 % syft cgr.dev/chainguard/jdk
 ✔ Loaded image                                                                                                                                                                                                                                                      cgr.dev/chainguard/jdk:latest
 ✔ Parsed image                                                                                                                                                                                                            sha256:c888584441cad85816725ae80ea8d874f623b67eab0240a24d6c62361111b2c4
 ✔ Cataloged packages              [54 packages]
NAME                    VERSION      TYPE
busybox                 1.36.1-r2    apk
ca-certificates-bundle  20230506-r0  apk
fontconfig-config       2.14.2-r2    apk
freetype                2.13.2-r1    apk
glibc                   2.38-r5      apk
glibc-locale-en         2.38-r5      apk
glibc-locale-posix      2.38-r5      apk
java-cacerts            20230106-r1  apk
java-common             0.1-r0       apk
jrt-fs                  21           java-archive
ld-linux                2.38-r5      apk
libbrotlicommon1        1.1.0-r0     apk
libbrotlidec1           1.1.0-r0     apk
libbz2-1                1.0.8-r5     apk
libcrypt1               2.38-r5      apk
libexpat1               2.5.0-r4     apk
libfontconfig1          2.14.2-r2    apk
libgcc                  13.2.0-r3    apk
libpng                  1.6.40-r0    apk
libstdc++               13.2.0-r3    apk
openjdk-21              21.35-r1     apk
openjdk-21-default-jdk  21.35-r1     apk
openjdk-21-default-jvm  21.35-r1     apk
openjdk-21-jre          21.35-r1     apk
openjdk-21-jre-base     21.35-r1     apk
wolfi-baselayout        20230201-r6  apk
zlib                    1.3-r0       apk

There's also a jre build:

% syft cgr.dev/chainguard/jre
 ✔ Pulled image
 ✔ Loaded image                                                                                                                                                                                                                                                      cgr.dev/chainguard/jre:latest
 ✔ Parsed image                                                                                                                                                                                                            sha256:6a519b42584aeaceaf029624f026c5f5157d5fda04c1843b58e13f00df3f71c4
 ✔ Cataloged packages              [45 packages]
NAME                    VERSION      TYPE
ca-certificates-bundle  20230506-r0  apk
fontconfig-config       2.14.2-r2    apk
freetype                2.13.2-r1    apk
glibc                   2.38-r5      apk
glibc-locale-en         2.38-r5      apk
glibc-locale-posix      2.38-r5      apk
java-cacerts            20230106-r1  apk
java-common             0.1-r0       apk
jrt-fs                  21           java-archive
ld-linux                2.38-r5      apk
libbrotlicommon1        1.1.0-r0     apk
libbrotlidec1           1.1.0-r0     apk
libbz2-1                1.0.8-r5     apk
libexpat1               2.5.0-r4     apk
libfontconfig1          2.14.2-r2    apk
libgcc                  13.2.0-r3    apk
libpng                  1.6.40-r0    apk
libstdc++               13.2.0-r3    apk
openjdk-21-default-jvm  21.35-r1     apk
openjdk-21-jre          21.35-r1     apk
openjdk-21-jre-base     21.35-r1     apk
wolfi-baselayout        20230201-r6  apk
zlib                    1.3-r0       apk

[loosebazooka]

Fundamentally it's easiest for us to do what debian does. I know we special case nodejs. And maybe we can do the same for Java -- we just don't right now. The wolfi images (or any other vendor) could fill the gap for you here until we find a Java 21 build.

[ding-ma]

I managed to build Java 21 and Python 3.12 distroless images based off of the debian SID release channel. I created my own debian package fetch script which generates a file similar to debian_archives.bzl.

Since I maintain a fork of this project at work, I can talk to my manager if I can contribute a part of my work back to this project. If I were to contribute back, we would need to modify the debian_package_manager so that it is able to fetch from the SID channel.

[loosebazooka]

I think we're just going to use temurin to do this: #1444

[loosebazooka]

So there should be some java21 images ready for testing. If you are using Java21, we would appreciate some feedback on if these are working for you.

gcr.io/distroless/java21-debian12 (amd64, arm64 and ppc64le). They are based on adoptium temurin java21 which does not seem to currently have an s390x build, so if you're using s390x you might be out of luck.

[swi-jared]

@loosebazooka I'm testing it now, and so far the only hiccup is that I had to upgrade jib which is probably good to do anyway (something about an unsupported image format). If there are any further hiccups I'll add them here.

Thanks!

[loosebazooka]

Glad to hear it's working, @swi-jared you probably should've run into that issue a while ago? Since we moved to oci media types (vs old docker media types) a few months ago.

[jochenberger]

I'm also running tests, looks good so far. :+1:

[swi-jared]

Glad to hear it's working, @swi-jared you probably should've run into that issue a while ago? Since we moved to oci media types (vs old docker media types) a few months ago.

This particular service has been using Temurin 17's base image, so we haven't run into this issue yet.

[apostoliska]

Hi all,

I am testing the java 21 distroless image and find some differences compared to Java 17 related to cacerts.

I am loading a custom trust store to /etc/ssl/certs/java/cacerts (through a volume) but is looks that it is ignored since the relevant symbolic link is missing.

Java 17:

/usr/lib/jvm/java-17-openjdk-amd64/lib/security $ ls -la total 8 drwxr-xr-x 2 root root 4096 Jul 24 09:57 . drwxr-xr-x 1 root root 4096 Jul 24 09:57 .. lrwxrwxrwx 1 root root 43 Jul 24 09:57 blocked.certs -> /etc/java-17-openjdk/security/blocked.certs lrwxrwxrwx 1 root root 27 Jul 24 09:57 cacerts -> /etc/ssl/certs/java/cacerts lrwxrwxrwx 1 root root 44 Jul 24 09:57 default.policy -> /etc/java-17-openjdk/security/default.policy lrwxrwxrwx 1 root root 52 Jul 24 09:57 public_suffix_list.dat -> /etc/java-17-openjdk/security/public_suffix_list.dat

Java 21:

/usr/lib/jvm/temurin21_jdk_amd64/lib/security $ ls -la total 436 drwxr-xr-x 2 root root 4096 Jan 1 2000 . drwxr-xr-x 5 root root 4096 Jan 1 2000 .. -r-xr-xr-x 1 root root 2488 Jan 1 2000 blocked.certs -r-xr-xr-x 1 root root 188019 Jan 1 2000 cacerts -r-xr-xr-x 1 root root 10657 Jan 1 2000 default.policy -r-xr-xr-x 1 root root 229405 Jan 1 2000 public_suffix_list.dat

[loosebazooka]

Yeah looks like temurin ships it's own certs. Probably makes sense to just use those? Or in your script replace those files?

[apostoliska]

Yes, I can try to mount my trust store to /usr/lib/jvm/temurin21_jdk_amd64/lib/security/cacerts.

I think this change should be noted. Using a custom trust store is quite frequent.

[loosebazooka]

Its not clear to me if we should add the symlinks and move the certs over to be consistent with the debian?

[jochenberger]

IMHO, it would be best to stay compatible with previous releases of java*-debian* so this can be a drop-in replacement.

[apostoliska]

Its not clear to me if we should add the symlinks and move the certs over to be consistent with the debian?

I believe having the symlinks would be preferable to simplify the upgrade (at least in my case). After all, openjdk 17 has its own cacerts as well and the symlinks were there.

[christopher-cudennec]

Hey everyone! Since #1444 is already merged: Will you also add gcr.io/distroless/java21-debian12 to the list of available images in your documentation? Thanks, Christopher

[loosebazooka]

Hrmm... Also wondering if we should use temurins shipped trust root or the debian one. Using temurins would simplify our code a bit.

[apostoliska]

I think that gcr.io/distroless/java17-debian12 used the cacerts shipped with Debian.

If that's the case, I believe the same should apply for java 21.

[loosebazooka]

Hr... that is the case, however, the debian java distributions also ship blocked.certs, default.policy and public_suffix_list.dat separately with the JRE install and we will be missing those. Using those could result in some mismatch?

It's looking like we should just use the temurin install as is (easier to maintain, and how temurin generally behaves). Java users will be expected to inject their custom roots directly into the jvm directory as documented in the Java README?

[apostoliska]

There is another issue that I am facing.

I am using the below images:

• gcr.io/distroless/java21-debian12:nonroot
• gcr.io/distroless/java21-debian12:debug-nonroot

I mount a custom cacerts to /usr/lib/jvm/temurin21_jdk_amd64/lib/security/cacerts . The application is performing an HTTPS request to a service that its certificated is loaded to the custom cacerts.

The results for each image are the following:

• debug-nonroot: No issue is observed
• nonroot: I get the below error. The custom cacerts appears to be ignored.

Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source) at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source) at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) at java.base/java.security.AccessController.doPrivileged(Unknown Source) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source) at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1647) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1493) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) ... 18 common frames omitted Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source) at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source) at java.base/sun.security.validator.Validator.validate(Unknown Source) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source) ... 30 common frames omitted Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source) at java.base/java.security.cert.CertPathBuilder.build(Unknown Source) ... 35 common frames omitted

The issue is resolved by running the app with the following parameter:

-Djavax.net.ssl.trustStore=/usr/lib/jvm/temurin21_jdk_amd64/lib/security/cacerts

However, adding the parameter should not be required as this is the default cacerts location. Additionally, there is a mismatch in the behavior between nonroot and debug-nonroot images. I find no other difference apart from the image tag.

Any ideas?

[ding-ma]

I'm still working on open sourcing the code. I'll submit PR once it is approved and you guys can decide which version to keep. The builds are based off of debian sid channel so it would be similar to the current java 17 builds.

@apostoliska, debug image contain java jdk whereas non-debug image does not.

I was wondering if we could make a jdk image without busybox. We would have the following:

1. gcr.io/distroless/java21-debian12:{latest,debug...}
2. gcr.io/distroless/java21-jdk-debian12:{latest,debug...}

The debug tag will only container busybox.

[loosebazooka]

@apostoliska I think @ding-ma answered your questions, but the detail is that regular images are jres, not jdks. So you need to use the jre path: /usr/lib/jvm/temurin21_jre_amd64/lib/security/cacerts But also maybe the names don't have to be different. Maybe both jre and jvm can just install to the same locaiton.

[pstackle]

I'll add myself as being in support of having separate images with the JDK versus the JRE. I had previously been operating under the assumption that the only difference between the debug and non-debug images was busybox being installed, so was surprised when I learned that the java debug image also used a JDK install instead of a JRE install.

I was wondering if we could make a jdk image without busybox.

My hope would be to also have a JRE image with busybox.