Enable to generate and package a CDS archive

[rmannibucau]

Hi,

CDS enables java application to boot way faster (in particular when any scanning/reflection is involved) so it would be neat to integrate it with JIB. It requires to launch 2-3 commands to generate the archive and then just bundle it and modify the jvm arguments.

https://blog.codefx.org/java/application-class-data-sharing/#Creating-A-JDK-Class-Data-Archive explains it quite well.

Since the classpath must be stable and not use folders nor wildcards, doing it in jib seems the most reliable and relevant IMHO.

Romain

[chanseokoh]

Never heard of CDS, but this does seem worth looking into it at some point. Thanks for the feedback! Does CDS only work with -jar my.jar or can work with the classpath launch style (-cp ... com.example.MyMain)?

[rmannibucau]

Wrote a quick post about it: https://rmannibucau.metawerx.net/post/java-class-data-sharing-docker-startup

Long story short, CDS is supported by any java app but classpath beginning must be stable and match the archive so jib can't use lib/*.jar for example.

I wrapped jib in a custom main and gain is ~30% on java 11 (i'm using zulu) on my app (CDI) so definitively worth enabling :).

[chanseokoh]

Thanks for the info. From which Java version can we use this feature?

jib can't use lib/*.jar for example.

@GoogleContainerTools/java-tools-build I remember the discussion that we can't get rid of the classpath wildcard libs/* until Java 9+ because of the max argument length limit or the max command line length (particularly short on Windows). ~~However, looks like modern Linux kernels support up to 2MB and seems like this is not an issue in practice. (We don't have to think about Windows.)~~

I have lost the confidence. (And who knows if Windows would matter if, e.g., running a Linux container on a Windows dev machine?)

[rmannibucau]

@chanseokoh think it was around java 10 (not sure commands are 100% the same in the first versions). These commands work well on java 11. BTW I would expect this feature to be off by default since it requires to execute docker commands so somebody enabling it would do it intentionally and classpath would fit the command line anyway I guess?

[chanseokoh]

FTR: https://cloud.google.com/run/docs/tips/java#appcds https://github.com/saturnism/jvm-helloworld-by-example/tree/master/helloworld-springboot-tomcat-shaded

[saturnism]

I took Jib for a spin to see how to do this mechanically (w/o automation). The basics premises are:

1. JDK arch/distribution needs to be the same
2. We need JARs - no exploded classpath, and no nested JARs. Just JARs listed in classpath.

It turns out that Jib's packaged containerization mode is a great fit to be able to produce just the JARs! The only issue is AppCDS's -cp can't take wildcard, so we need to list out individual JARs, which is discussed in #2733. Starting with Jib 2.7.0, this can be done by setting <container><expandClasspathDependencies>true.

I was experimenting w/ my Hello World Spring Boot App https://github.com/saturnism/jvm-helloworld-by-example/tree/master/helloworld-springboot-tomcat

1. Containerize w/ Packaged Mode, to local Docker daemon, and also need to use a debug base image so I can generate the classpath list using shell script.

   mvn package com.google.cloud.tools:jib-maven-plugin:2.7.0:dockerBuild \
     -Dimage=helloworld-experiment \
     -Djib.containerizingMode=packaged \
     -Djib.container.expandClasspathDependencies=true \
     -Djib.from.image=gcr.io/distroless/java-debian10:11-debug
   

2. Generate the class list, and the archive in the image

   # The Java CLASSPATH is the third element of the default image `ENTRYPOINT`
   # in the Jib-built image, e.g., "java -cp <...classpath...> com.example.MyMain".
   JIB_CLASSPATH=$( docker inspect helloworld-experiment --format '{{(index .Config.Entrypoint 2)}}' )
   
   docker run --entrypoint=sh --name=helloworld-experiment helloworld-experiment \
     -c "mkdir -p /app/appcds \
         && java -XX:DumpLoadedClassList=/app/appcds/classes.lst \
                 -cp '$JIB_CLASSPATH' \
                 com.example.helloworld.HelloworldApplication --appcds=true \
         && java -Xshare:dump \
                 -XX:SharedClassListFile=/app/appcds/classes.lst \
                 -XX:SharedArchiveFile=/app/appcds/archive.jsa \
                 -cp '$JIB_CLASSPATH'"
   

3. Commit the changes

   docker commit helloworld-experiment helloworld-experiment
   docker rm helloworld-experiment
   

4. Produce the new container image w/ a different entrypoint. Was hoping to use Jib CLI for this, but ran into some issues.

   # Produce the classpath again
   JIB_ENTRYPOINT='"/usr/bin/java","-Xshare:on","-XX:SharedArchiveFile=/app/appcds/archive.jsa","-cp","'${JIB_CLASSPATH}'","com.example.helloworld.HelloworldApplication"'
   
   cat << EOF > Dockerfile.appcds
   FROM helloworld-experiment
   
   CMD []
   ENTRYPOINT [ $JIB_ENTRYPOINT ]
   EOF
   
   docker build -f Dockerfile.appcds -t helloworld-experiment:appcds .
   

5. You can then run the container image w/ AppCDS

   docker run -ti --rm helloworld-experiment:appcds
   

[chanseokoh]

#2866 added the option jib.container.expandClasspathDependencies, and setting it to false will enumerate the dependency classpath (not yet released).

[chanseokoh]

@saturnism @rmannibucau @koeberlue @holledauer @olivierboudet @bric3 @guillaumeblaquiere @bilak we've released Jib 2.7.0 which added a new configuration option (jib.container.expandClasspathDependencies (Gradle) / <container><expandClasspathDependencies> (Maven)) that enables expanding classpath dependencies in the default java command for an image ENTRYPOINT. Turning on the option (off by default) will enumerate all the dependencies, which will match the dependency loading order in Maven or Gradle builds. For example, the ENTRYPOINT becomes

java ... -cp /app/resources:/app/classes:/app/libs/spring-boot-starter-web-2.0.3.RELEASE.jar:/app/libs/shared-library-0.1.0.jar:/app/libs/spring-boot-starter-json-2.0.3.RELEASE.jar:... com.example.Main

instead of the default

java ... -cp /app/resources:/app/classes:/app/libs/* com.example.Main

Expanding the dependency list will help the AppCDS use case above.

Note that an expanded dependency list can become very long in practice, and we are not sure if there may be a potential issue due to a long command line ("argument list too long" or "command line is too long").

As with other Jib configurations, this option can also be set through the system property (-Djib.container.expandClasspathDependencies=true|false).

[rmannibucau]

Does it work with extra classpath? Cds works with classpath prefix which must be expanded but end can stays a wildcard which helps to mount plugins. Would be great to have that feature without going with jibcore programmatic option.

[chanseokoh]

@rmannibucau no, Jib will just add the list of strings set by extraClasspath as-is, whether it contains a wildcard (*) or not. They are custom classpath, and it's not feasible for Jib to determine or enforce some order of expanding wildcards in custom classpath. (According to https://github.com/GoogleContainerTools/jib/issues/2733#issue-687396881, the loading order of * seems to depend on filesystems (and potentially JVMs)). So it's interesting that AppCDS can safely use a wildcard?

[rmannibucau]

@chanseokoh it is more about ensuring extra classpath is appended to the libs than prepended (recall it was prepended at some point - https://github.com/GoogleContainerTools/jib/pull/1642/files#diff-a5317ef6dce278f4451fa8e298358067261e7d10b3cca71c093673202ebb6d5cR276 ). If prepended it breaks cds, if appended it will keep CDS working well.

[chanseokoh]

Oh, now I understand. For AppCDS to work, it's enough for only some front portion of the entire classpath to be identical and it's fine to have a different classpath entries for the back portion, including using a wildcard, right?

Hmm... yeah, intentionally we prepend extraClasspath so that resources and classes from there take precedence. I wish it were easy to fix #894, which would have made extraClasspath obsolete. Maybe #894 could be supported with a new Jib extension?

[rmannibucau]

@chanseokoh well we can do anything with extensions but it kind of break using jib and using multiple extensions will quickly be hard so let's try to maybe keep it "core" until it is a specific feature? I see three simple options (in terms of usage and impl):

1. use a placeholder with known keywords: ${projectClasspath}:${extraLibs}
2. (preferred since easier for everybody to use and impl) add an enum PREPEND/APPEND in extractClasspath
3. (not directly linked to the order but more this issue) if expanded, extractClasspath goes at the end implicitly, this is more a workaround but works.

To have written several mains manipulating the entrypoint I'm really unhappy with this solution and it does not merge well with a concurrent extension doing the same so hope it hits jib-core/maven-plugin soon.

[chanseokoh]

Thanks for the input. Perhaps it makes sense to reduce the scope of #894 and enable simple keyword substitution only for <entrypoint> (after which extraClasspath can be deprecated).

Just in case, Jib extensions don't run concurrently but in the order they are defined.

[rmannibucau]

Yes in order but combining them is hard, it is easy to break previous one and in practise easier to use exec mvn plugin with jib-core :(.

[chanseokoh]

@saturnism I've updated your AppCDS demo using Jib 2.7.0 which has the option <container><expandClasspathDependencies> to expand the wildcard (*) in the classpath.

[chanseokoh]

@saturnism @rmannibucau @koeberlue @holledauer @olivierboudet @bric3 @guillaumeblaquiere @bilak @bademux @ykayacan @bilak @biro456 @zecit @chlund

Jib 3.1.1 is released and it creates two new JVM argument files inside an image, where they hold the computed classpath and the main class respectively. Although I haven't tried, I think this will greatly simplify the process to generate and use AppCDS. See the doc for more details. I'd like to try it myself and update the example above, but I'm not sure when I will find time for that.

[rmannibucau]

@chanseokoh hello, I fear it does not do anything since CDS is only about tuning the JAVA_OPTS base image env variable (_JAVA_OPTIONS often too) to enable or not the CDS archive once generated but all the missing part is the generation, not the enablement.

Side note: this "file" feature likely need the same for system properties to be useful, otherwise it is quite trivial to replace the entrypoint and use a wildcard for the classpath, main class not being not a challenge to switch in general ;).

[chanseokoh]

@rmannibucau to be fair, in the AppCDS context, I was mainly talking about the ugliness of this hack:

# The Java CLASSPATH is the third element of the default image `ENTRYPOINT`
# in the Jib-built image, e.g., "java -cp <...classpath...> com.example.MyMain".
JIB_CLASSPATH=$( docker inspect helloworld-experiment --format '{{(index .Config.Entrypoint 2)}}' )

Also even though you can always hardcode the classpath that Jib would use by default, but I never liked the idea, because the way it is computed is an internal detail that may change in newer versions (which turned out to be true now that Jib uses an argument file and with the expandClasspathDependencies option). People need a reliable and stable way to retrieve the classpath and the main class. Also using a wildcard makes the class loading order non-deterministic, so that's why Jib now always enumerates all the dependencies if Java 9+.

[rmannibucau]

@chanseokoh I see the point but this is still overridable from the yaml or CLI so being able to run some docker command before creating the final image is really the missing piece (or doing it in plain java since most needed steps are doable without running another java - but requires some internal knowledge IIRC). That said, thinking out loud, it can be another GoogleContainerTools project (experimental and kind of jib-core wrapper?).

[chanseokoh]

Thanks for the feedback! We did envision a grand plan to be able to use a container runtime, which we thought would also help with GraalVM native-image. Doesn't seem like we can reach that state anytime soon though. Another wrapper-like tool is a cool idea too.

[rmannibucau]

Geronimo arthur already handles graalvm wrapping jib, only cds remains ;)

[chanseokoh]

Ah, very interesting project. Thanks for the pointer.

[olivierboudet]

In my case, I can't have it to work but perhaps it is a JDK issue. I am using Jib 3.1.1 with a spring boot application. When running with flags -Xshare:dump -XX:SharedClassListFile=/tmp/cds/classes.lst -XX:SharedArchiveFile=/tmp/cds/application.jsa I encounter the following error :

Rewriting and linking classes: done
Error: non-empty directory '/app/resources'
Error: non-empty directory '/app/classes'
Hint: enable -Xlog:class+path=info to diagnose the failure
Error occurred during initialization of VM
Cannot have non-empty directory in paths

This is described in this issue but I am using OpenJDK 11.0.11 so it should be fixed.

[rmannibucau]

@olivierboudet maybe use packaged mode?

[olivierboudet]

@olivierboudet maybe use packaged mode?

You are totally right, it works with packaged mode but I don't see the point to use Jib without building layered images.

[rmannibucau]

@olivierboudet nothing prevents you to use layered images, put spring-boot stack in a first image (a project without spring app, use SpringApplication as fake main) then your app with the spring-boot stack as a provided dependencies and voilà. Side note being pakcage mode is not related to layers, I spoke of using target/yourapp.jar instead of resources/ and classes.