quarkus
quarkus copied to clipboard
Published
20 hours ago •
quarkusio
quarkusio
Build time performance regression and bigger native binaries when migrating from 3.5 to 3.6 or 3.7
Describe the bug
This was originally reported in https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/performance.20decrease.20with.20graalvm.2023.2E1.2E2.20-quarkus.203.2E5.20to.203.2E6 by @vsevel
hello, I am trying to upgrade one of our app that is running today with 3.2 I am seeing big perf decrease on the native build, specifically between quarkus 3.5 and 3.6 up to 3.5 (mandrel 23.0.3), I get native builds with around 10Gb of Peak RSS and augmentation time around 450 secs starting with 3.6 (mandrel 23.1.2), I get native builds with around 15Gb of Peak RSS and augmentation time around 900 secs
this seems related to the mandrel version change.
With Quarkus 3.5 and Mandrel 23.0.3
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on MANDREL 23.0.3.0 JDK 17.0.10+7
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --name build-native-JpspI quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17 -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Dlogging.initial-configurator.min-level=500 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dio.quarkus.caffeine.graalvm.recordStats=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 --features=io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature,io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature,io.quarkus.hibernate.validator.runtime.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:BuildOutputJSONFile=xx-5.1.1-SNAPSHOT-runner-build-output-stats.json --allow-incomplete-classpath -H:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback -H:+ReportExceptionStackTraces -J-Xmx20g -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:-UseServiceLoaderFeature -H:+StackTrace -J--add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/native-image\.properties --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/reflect-config\.json --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json xx-5.1.1-SNAPSHOT-runner -jar xx-5.1.1-SNAPSHOT-runner.jar
========================================================================================================================
GraalVM Native Image: Generating 'xx-5.1.1-SNAPSHOT-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
Warning: Feature class oracle.nativeimage.NativeImageFeature is annotated with the deprecated annotation @AutomaticFeature. Support for this annotation will be removed in a future version of GraalVM. Applications should register a feature using the option --features=oracle.nativeimage.NativeImageFeature
[1/8] Initializing... (40.5s @ 0.44GB)
Java version: 17.0.10+7, vendor version: Mandrel-23.0.3.0-Final
Graal compiler: optimization level: 2, target machine: x86-64-v3
C compiler: gcc (redhat, x86_64, 8.5.0)
Garbage collector: Serial GC (max heap size: 80% of RAM)
6 user-specific feature(s)
- com.oracle.svm.thirdparty.gson.GsonFeature
- io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature
- io.quarkus.hibernate.validator.runtime.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.hibernate.validator.internal.util.Version] categories
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
- io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
- oracle.nativeimage.NativeImageFeature
The bundle named: oracle.net.mesg.Message, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
The bundle named: oracle.net.mesg.Message, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
15:30:29,939 WARN [fre.dom] No XPath support is available. If you need it, add Apache Xalan or Jaxen as dependency.
[2/8] Performing analysis... [**********] (125.1s @ 3.84GB)
26,311 (77.92%) of 33,766 types reachable
49,848 (70.67%) of 70,540 fields reachable
170,497 (67.72%) of 251,765 methods reachable
9,987 types, 3,332 fields, and 51,258 methods registered for reflection
71 types, 93 fields, and 56 methods registered for JNI access
4 native libraries: dl, pthread, rt, z
[3/8] Building universe... (17.6s @ 4.14GB)
[4/8] Parsing methods... [****] (15.0s @ 3.62GB)
[5/8] Inlining methods... [****] (5.3s @ 4.28GB)
[6/8] Compiling methods... [*********] (80.9s @ 7.62GB)
[7/8] Layouting methods... [****] (13.4s @ 5.20GB)
[8/8] Creating image... [*****] (21.9s @ 6.63GB)
60.22MB (50.47%) for code area: 123,816 compilation units
58.71MB (49.20%) for image heap: 653,443 objects and 136 resources
410.26kB ( 0.34%) for other data
119.33MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
13.52MB java.base 13.26MB byte[] for code metadata
6.00MB svm.jar (Native Image) 6.89MB java.lang.Class
3.84MB xx-5.1.1-SNAPSHOT-runner.jar 6.31MB byte[] for java.lang.String
3.33MB io.fabric8.kubernetes-model-core-6.8.1.jar 5.90MB java.lang.String
2.18MB org.freemarker.freemarker-2.3.32.jar 4.99MB byte[] for general heap data
1.93MB c.f.jackson.core.jackson-databind-2.15.3.jar 4.67MB byte[] for reflection metadata
1.32MB modified-io.vertx.vertx-core-4.4.6.jar 2.25MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
1.28MB java.net.http 2.21MB com.oracle.svm.core.hub.DynamicHubCompanion
1.24MB io.fabric8.openshift-model-6.8.1.jar 1.22MB java.lang.String[]
627.07kB java.naming 953.16kB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
23.95MB for 265 more packages 9.87MB for 5682 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
HEAP: Set max heap for improved and more predictable memory usage.
CPU: Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
27.2s (8.2% of total time) in 92 GCs | Peak RSS: 10.41GB | CPU load: 4.18
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
/project/xx-5.1.1-SNAPSHOT-runner (executable)
/project/xx-5.1.1-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'xx-5.1.1-SNAPSHOT-runner' in 5m 23s.
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --entrypoint /bin/bash quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17 -c objcopy --strip-debug xx-5.1.1-SNAPSHOT-runner
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 388452ms
With Quarkus 3.6 and Mandrel 23.1.2
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on MANDREL 23.1.2.0 JDK 21.0.2+13-LTS
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --name build-native-ORizE quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Dlogging.initial-configurator.min-level=500 -J-Dio.quarkus.caffeine.graalvm.recordStats=true -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 --features=io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature,io.quarkus.hibernate.validator.runtime.DisableLoggingFeature,io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=xx-5.1.1-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions --strict-image-heap --allow-incomplete-classpath -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -J-Xmx20g -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/native-image\.properties --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/reflect-config\.json xx-5.1.1-SNAPSHOT-runner -jar xx-5.1.1-SNAPSHOT-runner.jar
Warning: Using a deprecated option --allow-incomplete-classpath from command line. Allowing an incomplete classpath is now the default. Use --link-at-build-time to report linking errors at image build time for a class or package.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where t...
========================================================================================================================
GraalVM Native Image: Generating 'xx-5.1.1-SNAPSHOT-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
Warning: Feature class oracle.nativeimage.NativeImageFeature is annotated with the deprecated annotation @AutomaticFeature. Support for this annotation will be removed in a future version of GraalVM. Applications should register a feature using the option --features=oracle.nativeimage.NativeImageFeature
[1/8] Initializing... (52.4s @ 0.42GB)
Java version: 21.0.2+13-LTS, vendor version: Mandrel-23.1.2.0-Final
Graal compiler: optimization level: 2, target machine: x86-64-v3
C compiler: gcc (redhat, x86_64, 8.5.0)
Garbage collector: Serial GC (max heap size: 80% of RAM)
6 user-specific feature(s):
- com.oracle.svm.thirdparty.gson.GsonFeature
- io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature
- io.quarkus.hibernate.validator.runtime.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.hibernate.validator.internal.util.Version] categories
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
- io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
- oracle.nativeimage.NativeImageFeature
------------------------------------------------------------------------------------------------------------------------
4 experimental option(s) unlocked:
- '-H:+AllowFoldMethods' (origin(s): command line)
- '-H:BuildOutputJSONFile' (origin(s): command line)
- '-H:-UseServiceLoaderFeature' (origin(s): command line)
- '-H:ReflectionConfigurationResources' (origin(s): 'META-INF/native-image/io.micrometer/micrometer-core/native-image.properties' in 'file:///project/lib/io.micrometer.micrometer-core-1.11.5.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
- 17.78GB of memory (70.9% of 25.06GB system memory, set via '-Xmx20g')
- 6 thread(s) (100.0% of 6 available processor(s), determined at start)
15:47:00,469 WARN [fre.dom] No XPath support is available. If you need it, add Apache Xalan or Jaxen as dependency.
[2/8] Performing analysis... [*****] (468.2s @ 9.26GB)
44,742 reachable types (93.8% of 47,723 total)
91,492 reachable fields (81.3% of 112,513 total)
403,198 reachable methods (84.6% of 476,837 total)
28,302 types, 57,105 fields, and 238,786 methods registered for reflection
69 types, 91 fields, and 56 methods registered for JNI access
4 native libraries: dl, pthread, rt, z
[3/8] Building universe... (26.1s @ 9.78GB)
[4/8] Parsing methods... [*****] (27.2s @ 10.15GB)
[5/8] Inlining methods... [****] (11.7s @ 9.41GB)
[6/8] Compiling methods... [************] (141.5s @ 5.30GB)
[7/8] Layouting methods... [******] (39.4s @ 7.28GB)
[8/8] Creating image... [********] (66.7s @ 8.28GB)
154.67MB (48.39%) for code area: 352,852 compilation units
164.56MB (51.48%) for image heap:1,238,079 objects and 200 resources
418.96kB ( 0.13%) for other data
319.63MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
18.46MB io.fabric8.openshift-model-operator-6.9.2.jar 48.89MB byte[] for code metadata
16.82MB svm.jar (Native Image) 28.81MB byte[] for java.lang.String
14.59MB io.fabric8.openshift-model-6.9.2.jar 22.70MB java.lang.Class
13.77MB java.base 19.13MB byte[] for reflection metadata
9.21MB io.fabric8.kubernetes-model-core-6.9.2.jar 11.04MB java.lang.String
4.83MB io.fabric8.openshift-model-monitoring-6.9.2.jar 10.87MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
4.80MB io.fabric8.openshift-model-hive-6.9.2.jar 3.75MB com.oracle.svm.core.hub.DynamicHubCompanion
4.20MB xx-5.1.1-SNAPSHOT-runner.jar 2.65MB java.lang.Object[]
2.98MB i.f.k.9.2.jar 2.31MB java.lang.String[]
2.78MB io.fabric8.openshift-model-operatorhub-6.9.2.jar 1.77MB byte[] for general heap data
59.56MB for 267 more packages 12.65MB for 5792 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
HEAP: Set max heap for improved and more predictable memory usage.
CPU: Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
129.8s (15.4% of total time) in 249 GCs | Peak RSS: 15.80GB | CPU load: 4.53
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
/project/xx-5.1.1-SNAPSHOT-runner (executable)
/project/xx-5.1.1-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'xx-5.1.1-SNAPSHOT-runner' in 13m 57s.
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --entrypoint /bin/bash quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21 -c objcopy --strip-debug xx-5.1.1-SNAPSHOT-runner
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 933775ms
Expected behavior
Build times and binary size are expected to be comparable between 3.5, 3.6 and 3.7 given no other changes happened in the application code.
Actual behavior
The user observes significant increases in build execution times, memory usage, and binary size.
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
Mandrel or GraalVM version (if different from Java)
No response
Quarkus version or git rev
No response
Build tool (ie. output of mvnw --version or gradlew --version)
No response
Additional information
No response
You added a link to a Zulip discussion, please make sure the description of the issue is comprehensive and doesn't require accessing Zulip
This message is automatically generated by a bot.
![quarkus-bot[bot] avatar](https://avatars.githubusercontent.com/in/90234?v=4 "Published by a pub.dev verified publisher"){kind=small}
/cc @Karm (mandrel), @galderz (mandrel), @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift)
Looking at the provided build output one can already see some things::
- The time spent in the analysis is way longer in 23.1 than in 23.0 and that's reflected in the reachable types. What's interesting is that not only have the reachable types increased as a % of the total types, but the total reachable types also increase (same for fields and methods).
- There is a big increase in the types registered for reflection.
Note that these differences do not match the ones we observed in https://quarkus.io/blog/mandrel-23-1-image-size-increase/
It looks like in Q 3.6 we are building double the code...
If I understand correctly the way this was tested is by building the exact same code with Quarkus 3.5 and then with Quarkus 3.6, which means that you use both a different set of dependencies and a different builder image. Could you please run another build with Quarkus 3.6 but this time with the jdk-17 builder image? i.e. passing -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17
This will allow us to see if the main difference is due to the different Mandrel builder image or due to some Quarkus change.
It looks like what you are seeing is also related to https://github.com/quarkusio/quarkus/issues/37142
You might want to give https://github.com/quarkusio/quarkus/pull/37278 a try:
This PR allows for the exclusion of openshift-model-hive, openshift-model-miscellaneous, openshift-model-operator in native mode, which will already downsize the resulting binary considerably.
Indeed running with Quarkus 3.7.1 and Mandrel 23.0.3 gives similar results, indicating that this is not caused by the Mandrel update, but by some dependency change (e.g. "in Quarkus 3.5.3, we are using io.fabric8 6.8.1, and 6.9.2 in 3.6"):
$ mvn clean package -o -DskipTests -Dnative -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on MANDREL 23.0.3.0 JDK 17.0.10+7
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --name build-native-KGynW quay.io/quarkus/ub
i-quarkus-mandrel-builder-image:jdk-17 -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dio.quarkus.caffein
e.graalvm.recordStats=true -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetec
tion.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.hibernate.validator.runtime.DisableLoggingFeature,io.quarkus.r
untime.graal.DisableLoggingFeature,io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/
java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:BuildOutputJSONFile=xx-5.1.1-SNAPSHOT-runner-build-output-stats.json --allow-incomplete-classpath -H
:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback -H:+ReportExceptionStackTraces -J-Xmx20g -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:-UseServiceLoaderFeature -H:+StackTrace -J-
-add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/native-image\.properties --exclude-config com\.oracle\.database\.jdbc /META-INF/native-imag
e/reflect-config\.json --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-
handler/generated/handlers/reflect-config\.json xx-5.1.1-SNAPSHOT-runner -jar xx-5.1.1-SNAPSHOT-runner.jar
Warning: Using a deprecated option --allow-incomplete-classpath from command line. Allowing an incomplete classpath is now the default. Use --link-at-build-time to report linking errors at image build time for a class or package.
========================================================================================================================
GraalVM Native Image: Generating 'xx-5.1.1-SNAPSHOT-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
Warning: Feature class oracle.nativeimage.NativeImageFeature is annotated with the deprecated annotation @AutomaticFeature. Support for this annotation will be removed in a future version of GraalVM. Applications should register
a feature using the option --features=oracle.nativeimage.NativeImageFeature
...
[1/8] Initializing... (54.9s @ 0.46GB)
Java version: 17.0.10+7, vendor version: Mandrel-23.0.3.0-Final
Graal compiler: optimization level: 2, target machine: x86-64-v3
C compiler: gcc (redhat, x86_64, 8.5.0)
Garbage collector: Serial GC (max heap size: 80% of RAM)
6 user-specific feature(s)
- com.oracle.svm.thirdparty.gson.GsonFeature
- io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature
- io.quarkus.hibernate.validator.runtime.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.hibernate.validator.internal.util.Version] categories
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
- io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
- oracle.nativeimage.NativeImageFeature
The bundle named: oracle.net.mesg.Message, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
The bundle named: oracle.net.mesg.Message, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
17:17:51,730 WARN [fre.dom] No XPath support is available. If you need it, add Apache Xalan or Jaxen as dependency.
[2/8] Performing analysis... [*****] (865.3s @ 8.90GB)
44,303 (94.03%) of 47,115 types reachable
91,352 (81.78%) of 111,702 fields reachable
402,379 (84.68%) of 475,176 methods reachable
28,167 types, 57,447 fields, and 240,611 methods registered for reflection
71 types, 93 fields, and 56 methods registered for JNI access
4 native libraries: dl, pthread, rt, z
[3/8] Building universe... (82.7s @ 8.96GB)
GC warning: 47.0s spent in 5 GCs during the last stage, taking up 56.49% of the time.
Please ensure more than 14.26GB of memory is available for Native Image
to reduce GC overhead and improve image build time.
[4/8] Parsing methods... [******] (44.4s @ 10.07GB)
[5/8] Inlining methods... [****] (13.3s @ 8.63GB)
[6/8] Compiling methods... [***************] (240.5s @ 5.06GB)
[7/8] Layouting methods... [******] (44.2s @ 5.89GB)
[8/8] Creating image... [*******] (55.8s @ 6.98GB)
152.88MB (50.89%) for code area: 353,694 compilation units
147.16MB (48.98%) for image heap:1,233,195 objects and 138 resources
410.15kB ( 0.13%) for other data
300.43MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
18.32MB io.fabric8.openshift-model-operator-6.10.0.jar 31.86MB byte[] for code metadata
16.92MB svm.jar (Native Image) 22.61MB java.lang.Class
14.28MB io.fabric8.openshift-model-6.10.0.jar 22.18MB byte[] for java.lang.String
13.53MB java.base 19.25MB byte[] for reflection metadata
9.11MB io.fabric8.kubernetes-model-core-6.10.0.jar 10.97MB java.lang.String
4.70MB io.fabric8.openshift-model-monitoring-6.10.0.jar 10.95MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
4.68MB io.fabric8.openshift-model-hive-6.10.0.jar 8.33MB byte[] for general heap data
4.66MB xx-5.1.1-SNAPSHOT-runner.jar 3.72MB com.oracle.svm.core.hub.DynamicHubCompanion
2.90MB i.f.k.10.0.jar 2.58MB java.lang.Object[]
/project/xx-5.1.1-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'xx-5.1.1-SNAPSHOT-runner' in 23m 26s.
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --entrypoint /bin/bash quay.io/quarkus/ubi-q
uarkus-mandrel-builder-image:jdk-17 -c objcopy --strip-debug xx-5.1.1-SNAPSHOT-runner
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 1527193ms
[INFO]
[INFO] --- build-helper-maven-plugin:3.0.0:attach-artifact (attach-artifacts) @ xx ---
[INFO] Skip attaching artifacts
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26:21 min
[INFO] Finished at: 2024-02-07T18:38:06+01:00
[INFO] ------------------------------------------------------------------------
@vsevel also reported that:
I was able to get some savings with the referenced exclusions (Quarkus 3.7.1 ):
75.9s (11.4% of total time) in 148 GCs | Peak RSS: 12.05GB | CPU load: 4.50 ... Finished generating 'x-5.1.1-SNAPSHOT-runner' in 10m 56s.better than without the exclusion, still not as good compared to 3.5 :
27.2s (8.2% of total time) in 92 GCs | Peak RSS: 10.41GB | CPU load: 4.18 ------------------------------------------------------------------------------------------------------------------------ Produced artifacts: /project/xx-5.1.1-SNAPSHOT-runner (executable) /project/xx-5.1.1-SNAPSHOT-runner-build-output-stats.json (build_info) ======================================================================================================================== Finished generating 'xx-5.1.1-SNAPSHOT-runner' in 5m 23s.I am going to try to see if I can add more exclusions. still I am very much surprised about the impact additional libraries can have on the build.
FWIW, I know that at some point a Kubernetes Client update caused some problems as the amount of code was massive.
I am continuing trying to get rid of as many models as possible. it is a pain, because some missing models appear at native runtime only. so it is a lengthy process.
FWIW, I know that at some point a Kubernetes Client update caused some problems as the amount of code was massive.
Correct, that was https://github.com/quarkusio/quarkus/issues/37142
with the latest exclusions in 3.7.1 I get on our CI:
49.4s (14.7% of total time) in 171 GCs | Peak RSS: 11.30GB | CPU load: 6.04
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 5m 35s.
on a test branch for 3.5 I get:
23.4s (10.0% of total time) in 115 GCs | Peak RSS: 8.32GB | CPU load: 5.77
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 3m 53s.
our baseline on 3.2 (master) :
30.2s (13.2% of total time) in 167 GCs | Peak RSS: 5.83GB | CPU load: 5.91
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 3m 48s.
the increase of memory is scary. our build machines are going to have a hard time coping with processes 11Gb of RAM.
here is the exclusions I was able to do:
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-openshift-client</artifactId>
<exclusions>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-hive</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-miscellaneous</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-operator</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-clusterautoscaling</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-operatorhub</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-machine</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-whereabouts</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-storageversionmigrator</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-tuned</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-console</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-machineconfig</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>openshift-model-installer</artifactId>
</exclusion>
</exclusions>
</dependency>
and
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>certmanager-client</artifactId>
<exclusions>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-resource</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-rbac</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-admissionregistration</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-autoscaling</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-coordination</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-discovery</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-events</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-flowcontrol</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-metrics</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-policy</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-scheduling</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-storageclass</artifactId>
</exclusion>
<exclusion>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-node</artifactId>
</exclusion>
</exclusions>
</dependency>
even with all that cleanup, this shows a signifiant increase in memory and build time. what else can I do?
3.5:
2024.02.07 12:58:38 [2/8] Performing analysis... [**********] (89.6s @ 3.25GB)
2024.02.07 12:58:38 25,787 (77.52%) of 33,265 types reachable
2024.02.07 12:58:38 49,039 (70.80%) of 69,265 fields reachable
2024.02.07 12:58:38 168,338 (67.65%) of 248,837 methods reachable
2024.02.07 12:58:38 9,796 types, 2,542 fields, and 51,144 methods registered for reflection
2024.02.07 12:58:38 71 types, 93 fields, and 56 methods registered for JNI access
2024.02.07 12:58:38 4 native libraries: dl, pthread, rt, z
2024.02.07 12:58:48 [3/8] Building universe... (12.8s @ 3.43GB)
2024.02.07 12:58:58 [4/8] Parsing methods... [***] (10.7s @ 3.57GB)
2024.02.07 12:59:06 [5/8] Inlining methods... [****] (6.8s @ 2.94GB)
2024.02.07 13:00:14 [6/8] Compiling methods... [*******] (57.7s @ 5.28GB)
2024.02.07 13:00:19 [7/8] Layouting methods... [****] (14.7s @ 3.91GB)
2024.02.07 13:00:41 [8/8] Creating image... [****] (20.2s @ 5.33GB)
2024.02.07 13:00:41 59.47MB (50.26%) for code area: 122,614 compilation units
2024.02.07 13:00:41 58.45MB (49.40%) for image heap: 644,721 objects and 136 resources
2024.02.07 13:00:41 408.55kB ( 0.34%) for other data
2024.02.07 13:00:41 118.31MB in total
2024.02.07 13:00:41 ------------------------------------------------------------------------------------------------------------------------
2024.02.07 13:00:41 Top 10 origins of code area: Top 10 object types in image heap:
2024.02.07 13:00:41 13.52MB java.base 13.09MB byte[] for code metadata
2024.02.07 13:00:41 5.99MB svm.jar (Native Image) 6.76MB java.lang.Class
2024.02.07 13:00:41 3.84MB ocpdeploy-5.1.1-SNAPSHOT-runner.jar 6.22MB byte[] for java.lang.String
2024.02.07 13:00:41 3.33MB io.fabric8.kubernetes-model-core-6.8.1.jar 5.81MB java.lang.String
2024.02.07 13:00:41 2.18MB org.freemarker.freemarker-2.3.32.jar 4.92MB byte[] for general heap data
2024.02.07 13:00:41 1.93MB c.f.jackson.core.jackson-databind-2.15.3.jar 4.61MB byte[] for reflection metadata
2024.02.07 13:00:41 1.32MB modified-io.vertx.vertx-core-4.4.6.jar 2.25MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
2024.02.07 13:00:41 1.28MB java.net.http 2.16MB com.oracle.svm.core.hub.DynamicHubCompanion
2024.02.07 13:00:41 1.23MB io.fabric8.openshift-model-6.8.1.jar 1.21MB java.lang.String[]
2024.02.07 13:00:41 627.05kB java.naming 934.53kB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
2024.02.07 13:00:41 23.24MB for 264 more packages 9.79MB for 5560 more object types
3.7
2024.02.08 17:24:43 [2/8] Performing analysis... [*****] (138.7s @ 5.92GB)
2024.02.08 17:24:43 32,715 reachable types (91.6% of 35,730 total)
2024.02.08 17:24:43 62,938 reachable fields (75.3% of 83,560 total)
2024.02.08 17:24:43 270,083 reachable methods (78.6% of 343,567 total)
2024.02.08 17:24:43 16,500 types, 29,030 fields, and 133,756 methods registered for reflection
2024.02.08 17:24:43 69 types, 91 fields, and 56 methods registered for JNI access
2024.02.08 17:24:43 4 native libraries: dl, pthread, rt, z
2024.02.08 17:24:55 [3/8] Building universe... (22.2s @ 6.12GB)
2024.02.08 17:25:07 [4/8] Parsing methods... [***] (11.4s @ 4.77GB)
2024.02.08 17:25:14 [5/8] Inlining methods... [****] (6.3s @ 7.43GB)
2024.02.08 17:26:36 [6/8] Compiling methods... [*********] (80.0s @ 4.02GB)
2024.02.08 17:27:03 [7/8] Layouting methods... [*****] (27.5s @ 7.24GB)
2024.02.08 17:27:35 [8/8] Creating image... [*****] (31.9s @ 4.55GB)
2024.02.08 17:27:35 107.24MB (50.12%) for code area: 224,347 compilation units
2024.02.08 17:27:35 106.34MB (49.69%) for image heap: 901,740 objects and 200 resources
2024.02.08 17:27:35 417.65kB ( 0.19%) for other data
2024.02.08 17:27:35 213.99MB in total
2024.02.08 17:27:35 ------------------------------------------------------------------------------------------------------------------------
2024.02.08 17:27:35 Top 10 origins of code area: Top 10 object types in image heap:
2024.02.08 17:27:35 14.59MB io.fabric8.openshift-model-6.10.0.jar 33.65MB byte[] for code metadata
2024.02.08 17:27:35 13.77MB java.base 17.27MB byte[] for java.lang.String
2024.02.08 17:27:35 10.05MB svm.jar (Native Image) 11.33MB java.lang.Class
2024.02.08 17:27:35 9.33MB io.fabric8.kubernetes-model-core-6.10.0.jar 10.98MB byte[] for reflection metadata
2024.02.08 17:27:35 4.83MB io.fabric8.openshift-model-monitoring-6.10.0.jar 8.13MB java.lang.String
2024.02.08 17:27:35 4.75MB ocpdeploy-5.1.1-SNAPSHOT-runner.jar 6.12MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
2024.02.08 17:27:35 2.98MB i.f.k.10.0.jar 2.75MB com.oracle.svm.core.hub.DynamicHubCompanion
2024.02.08 17:27:35 2.87MB io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar 1.70MB java.lang.Object[]
2024.02.08 17:27:35 2.24MB org.freemarker.freemarker-2.3.32.jar 1.70MB java.lang.String[]
2024.02.08 17:27:35 2.21MB io.fabric8.certmanager-model-v1alpha2-6.10.0.jar 1.49MB byte[] for general heap data
2024.02.08 17:27:35 37.92MB for 245 more packages 11.22MB for 5683 more object types
3.2
2024.02.07 22:55:47 [2/8] Performing analysis... [*********] (90.9s @ 2.90GB)
2024.02.07 22:55:47 26,276 (73.80%) of 35,603 types reachable
2024.02.07 22:55:47 49,003 (70.23%) of 69,779 fields reachable
2024.02.07 22:55:47 167,491 (64.73%) of 258,751 methods reachable
2024.02.07 22:55:47 10,020 types, 2,067 fields, and 48,594 methods registered for reflection
2024.02.07 22:55:47 71 types, 93 fields, and 56 methods registered for JNI access
2024.02.07 22:55:47 4 native libraries: dl, pthread, rt, z
2024.02.07 22:55:58 [3/8] Building universe... (12.5s @ 3.39GB)
2024.02.07 22:56:08 [4/8] Parsing methods... [***] (11.0s @ 2.04GB)
2024.02.07 22:56:13 [5/8] Inlining methods... [****] (4.1s @ 1.91GB)
2024.02.07 22:57:09 [6/8] Compiling methods... [*******] (55.3s @ 3.86GB)
2024.02.07 22:57:25 [7/8] Layouting methods... [****] (14.6s @ 2.77GB)
2024.02.07 22:57:43 [8/8] Creating image... [****] (18.8s @ 2.92GB)
2024.02.07 22:57:43 59.40MB (50.50%) for code area: 120,579 compilation units
2024.02.07 22:57:43 57.81MB (49.16%) for image heap: 645,267 objects and 134 resources
2024.02.07 22:57:43 412.78kB ( 0.34%) for other data
2024.02.07 22:57:43 117.61MB in total
2024.02.07 22:57:43 ------------------------------------------------------------------------------------------------------------------------
2024.02.07 22:57:43 Top 10 origins of code area: Top 10 object types in image heap:
2024.02.07 22:57:43 13.61MB java.base 13.10MB byte[] for code metadata
2024.02.07 22:57:43 6.02MB svm.jar (Native Image) 6.94MB java.lang.Class
2024.02.07 22:57:43 3.76MB ocpdeploy-5.1.1-SNAPSHOT-runner.jar 6.26MB byte[] for java.lang.String
2024.02.07 22:57:43 2.97MB io.fabric8.kubernetes-model-core-6.7.2.jar 5.83MB java.lang.String
2024.02.07 22:57:43 2.18MB org.freemarker.freemarker-2.3.32.jar 4.49MB byte[] for general heap data
2024.02.07 22:57:43 1.93MB c.f.jackson.core.jackson-databind-2.15.2.jar 4.43MB byte[] for reflection metadata
2024.02.07 22:57:43 1.30MB modified-io.vertx.vertx-core-4.4.4.jar 2.21MB com.oracle.svm.core.hub.DynamicHubCompanion
2024.02.07 22:57:43 1.28MB java.net.http 2.13MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
2024.02.07 22:57:43 1.14MB io.fabric8.openshift-model-6.7.2.jar 1.21MB java.lang.String[]
2024.02.07 22:57:43 629.42kB java.naming 944.41kB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
2024.02.07 22:57:43 23.58MB for 270 more packages 9.64MB for 5594 more object types
tests on 3.6 and 3.7 were run with a large xmx (20Gb). I reran the tests with the original xmx=7g that we have on master, and it changes the numbers I gave earlier:
3.7
55.1s (15.7% of total time) in 283 GCs | Peak RSS: 7.51GB | CPU load: 6.00
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 5m 50s.
3.6
57.8s (16.6% of total time) in 308 GCs | Peak RSS: 7.21GB | CPU load: 5.38
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 5m 46s.
baseline 3.2
30.2s (13.2% of total time) in 167 GCs | Peak RSS: 5.83GB | CPU load: 5.91
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 3m 48s.
not perfect, but memory looks far better.
even with all that cleanup, this shows a significant increase in memory and build time. what else can I do?
I'm not 100% sure, but I'd say that with those exclusions, the client models shouldn't be accountable for the increase in size.
Take the kubernetes-model-core as an example (which is fairly stable).
The original jar size vs the reported origins of code area:
- 3.2 / 6.7.2
4907714:2.97MB - 3.5 / 6.8.1
3665233:3.33MB - 3.7 / 6.10.0
3631645:9.33MB
Even when the original jar file is reducing its size (we did split it up at some point), the resulting size accounted for these types keeps increasing.
@vsevel looking at the 3.5 vs 3.7 build outputs (I am assuming you are using Mandrel 23.0 in 3.5 and Mandrel 23.1 in 3.7) I see that both the code area and the heap size increased.
For the code area increase (by ~50MB) the key contributors appear to be:
1. io.fabric8.openshift-model-6.10.0.jar (is ~13MB larger than it used to be in 6.8.1)
2. svm.jar (is ~4MB larger than it used to in Mandrel 23.0)
3. io.fabric8.kubernetes-model-core-6.10.0.jar (is ~6MB larger than it used to be in 6.8.1)
4. io.fabric8.openshift-model-monitoring-6.10.0.jar (brings at least 3MB extra)
5. io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar (brings at least 2MB extra)
6. ...
For the heap area increase (by ~50MB) the key contributors appear to be: 1. 20MB of extra metadata (part of this is explained in https://quarkus.io/blog/mandrel-23-1-image-size-increase/) 2. 11MB of extra String values 3. 6MB of extra metadata used for reflection (This is caused by registering more things for reflection) 4. 5MB of extra Classes 5. ...
All the above indicate that more code has become reachable and is leading in the build process to be more resource hungry and to take longer. It also results in a larger binary in the end. I think that at this point there is not much you can do on the application side, but we also can't to much without more info.
It would be great if you could share a reproducer with us, but if that's not possible it would also help if you could run a 3.5 and a 3.7 build and a quick run afterwards with the following parameters and share the output files with us.
parameters
-Dquarkus.native.additional-build-args=-R:+DumpHeapAndExit,-H:+DashboardAll,-H:+DashboardJson,-H:DashboardDump=path/to/dashboard.json
-Dquarkus.native.enable-reports
interesting files
target/your-app-native-image-sources/your-app-build-output-stats.json(created at build time)your-app.hprof(created after running the native executable of your app)target/your-app-native-image-sources/reports(this is a directory please include all files)dashboard.json.dump(created at build time)
@vsevel I forgot to mention that in order to reduce the moving parts to the minimum it would be better to build only using Mandrel 23.0 for the time being.
One way to identify more about what is increasing build time RSS would be to generate heap dumps during build and see which object types/how many object instances are being held on to. This might indicate which component of GraalVM Native is blowing up.
Heap dumps might also indicate whether the RSS increase is down to a GraalVM/Mandrel change or -- as looks more likely -- a Quarkus change. If you interpolate between old and new configs by generating heap dumps with the Quarkus 3.7.1 and Mandrel 23.0 then that might further clarify which of the two components is responsible (could still be both).
I have run some quick tests on a sample operator using QOSDK 6.3.x which targets Quarkus 3.2.10 and QOSDK main with Quarkus 3.7.3:
3.2.10:
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on native-image 21.0.2 2024-01-16
GraalVM Runtime Environment GraalVM CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30)
Substrate VM GraalVM CE 21.0.2+13.1 (build 21.0.2+13, serial gc)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /Users/claprun/.sdkman/candidates/java/21.0.2-graalce/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=FR -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner-build-output-stats.json -H:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+ReportExceptionStackTraces -H:-AddAllCharsets --enable-url-protocols=http,https -H:-UseServiceLoaderFeature -J--add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner -jar quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner.jar
Warning: The option '-H:+AllowFoldMethods' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:-UseServiceLoaderFeature' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner-build-output-stats.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.netty/netty-transport/reflection-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
WARNING: Unknown module: org.graalvm.sdk specified to --add-exports
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/8] Initializing... (12.3s @ 0.35GB)
Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
Graal compiler: optimization level: 2, target machine: armv8-a
C compiler: cc (apple, arm64, 15.0.0)
Garbage collector: Serial GC (max heap size: 80% of RAM)
3 user-specific feature(s):
- com.oracle.svm.thirdparty.gson.GsonFeature
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
- io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
------------------------------------------------------------------------------------------------------------------------
4 experimental option(s) unlocked:
- '-H:+AllowFoldMethods' (origin(s): command line)
- '-H:BuildOutputJSONFile' (origin(s): command line)
- '-H:-UseServiceLoaderFeature' (origin(s): command line)
- '-H:ReflectionConfigurationResources' (origin(s): 'META-INF/native-image/io.micrometer/micrometer-core/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-native-image-source-jar/lib/io.micrometer.micrometer-core-1.11.1.jar', 'META-INF/native-image/io.netty/netty-transport/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-native-image-source-jar/lib/io.netty.netty-transport-4.1.100.Final.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
- 26.49GB of memory (41.4% of 64.00GB system memory, determined at start)
- 10 thread(s) (100.0% of 10 available processor(s), determined at start)
java.lang.IllegalAccessError: class io.quarkus.runner.Feature (in unnamed module @0x6ab50d1c) cannot access class org.graalvm.nativeimage.impl.RuntimeClassInitializationSupport (in module org.graalvm.nativeimage) because module org.graalvm.nativeimage does not export org.graalvm.nativeimage.impl to unnamed module @0x6ab50d1c
at io.quarkus.runner.Feature.runtimeReinitializedClasses(Unknown Source)
at io.quarkus.runner.Feature.beforeAnalysis(Unknown Source)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:773)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:773)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
java.lang.IllegalAccessError: class io.quarkus.runner.Feature (in unnamed module @0x6ab50d1c) cannot access class org.graalvm.nativeimage.impl.RuntimeClassInitializationSupport (in module org.graalvm.nativeimage) because module org.graalvm.nativeimage does not export org.graalvm.nativeimage.impl to unnamed module @0x6ab50d1c
at io.quarkus.runner.Feature.runtimeReinitializedClasses(Unknown Source)
at io.quarkus.runner.Feature.beforeAnalysis(Unknown Source)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:773)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:773)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
java.lang.IllegalAccessError: class io.quarkus.runner.Feature (in unnamed module @0x6ab50d1c) cannot access class org.graalvm.nativeimage.impl.RuntimeClassInitializationSupport (in module org.graalvm.nativeimage) because module org.graalvm.nativeimage does not export org.graalvm.nativeimage.impl to unnamed module @0x6ab50d1c
at io.quarkus.runner.Feature.runtimeReinitializedClasses(Unknown Source)
at io.quarkus.runner.Feature.beforeAnalysis(Unknown Source)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:773)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:773)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
java.lang.IllegalAccessError: class io.quarkus.runner.Feature (in unnamed module @0x6ab50d1c) cannot access class org.graalvm.nativeimage.impl.RuntimeClassInitializationSupport (in module org.graalvm.nativeimage) because module org.graalvm.nativeimage does not export org.graalvm.nativeimage.impl to unnamed module @0x6ab50d1c
at io.quarkus.runner.Feature.runtimeReinitializedClasses(Unknown Source)
at io.quarkus.runner.Feature.beforeAnalysis(Unknown Source)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:773)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:773)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
[2/8] Performing analysis... [16:04:11,601 WARN [io.net.res.dns.DnsServerAddressStreamProviders] Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'
*********] (25.8s @ 2.03GB)
17,190 reachable types (69.7% of 24,678 total)
33,054 reachable fields (70.9% of 46,590 total)
122,784 reachable methods (63.3% of 193,960 total)
7,173 types, 871 fields, and 40,747 methods registered for reflection
61 types, 61 fields, and 55 methods registered for JNI access
5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
[3/8] Building universe... (3.9s @ 2.32GB)
[4/8] Parsing methods... [**] (2.7s @ 2.95GB)
[5/8] Inlining methods... [***] (1.2s @ 2.55GB)
[6/8] Compiling methods... [****] (16.7s @ 3.16GB)
[7/8] Layouting methods... [***] (5.4s @ 2.30GB)
[8/8] Creating image... [***] (8.0s @ 3.78GB)
40.02MB (46.83%) for code area: 89,175 compilation units
44.47MB (52.04%) for image heap: 447,396 objects and 56 resources
992.39kB ( 1.13%) for other data
85.46MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
11.78MB java.base 13.50MB byte[] for code metadata
4.76MB svm.jar (Native Image) 6.85MB byte[] for java.lang.String
2.87MB io.fabric8.kubernetes-model-core-6.7.2.jar 4.38MB java.lang.Class
1.80MB c.f.jackson.core.jackson-databind-2.15.2.jar 4.06MB java.lang.String
1.37MB q.3.6-SNAPSHOT-runner.jar 3.52MB byte[] for reflection metadata
1.24MB io.fabric8.openshift-model-6.7.2.jar 1.78MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
1.10MB modified-io.vertx.vertx-core-4.4.6.jar 1.44MB com.oracle.svm.core.hub.DynamicHubCompanion
1.08MB org.bouncycastle.bcprov-jdk18on-1.74.jar 860.73kB byte[] for general heap data
562.08kB io.fabric8.openshift-model-operator-6.7.2.jar 846.55kB java.lang.String[]
561.55kB com.fasterxml.jackson.core.jackson-core-2.15.2.jar 754.49kB java.lang.Object[]
12.49MB for 134 more packages 6.53MB for 3534 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
INIT: Adopt '--strict-image-heap' to prepare for the next GraalVM release.
HEAP: Set max heap for improved and more predictable memory usage.
CPU: Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
6.5s (8.3% of total time) in 103 GCs | Peak RSS: 6.32GB | CPU load: 5.44
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
/Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner (executable)
/Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner' in 1m 17s.
3.7.3:
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GRAALVM 23.1 JDK 21.0.2+13-jvmci-23.1-b30
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /Users/claprun/.sdkman/candidates/java/21.0.2-graalce/bin/native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=FR -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions --strict-image-heap -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner -jar quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner.jar
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/8] Initializing... (8.0s @ 0.43GB)
Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
Graal compiler: optimization level: 2, target machine: armv8-a
C compiler: cc (apple, arm64, 15.0.0)
Garbage collector: Serial GC (max heap size: 80% of RAM)
3 user-specific feature(s):
- com.oracle.svm.thirdparty.gson.GsonFeature
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
- io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
------------------------------------------------------------------------------------------------------------------------
3 experimental option(s) unlocked:
- '-H:+AllowFoldMethods' (origin(s): command line)
- '-H:BuildOutputJSONFile' (origin(s): command line)
- '-H:-UseServiceLoaderFeature' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
- 26.49GB of memory (41.4% of 64.00GB system memory, determined at start)
- 10 thread(s) (100.0% of 10 available processor(s), determined at start)
[2/8] Performing analysis... [16:21:48,585 WARN [io.net.res.dns.DnsServerAddressStreamProviders] Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'
*****] (191.1s @ 8.51GB)
34,517 reachable types (94.8% of 36,416 total)
73,068 reachable fields (84.1% of 86,892 total)
339,522 reachable methods (86.5% of 392,604 total)
24,339 types, 51,922 fields, and 216,678 methods registered for reflection
61 types, 61 fields, and 55 methods registered for JNI access
5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
[3/8] Building universe... (7.2s @ 11.65GB)
[4/8] Parsing methods... [***] (6.5s @ 7.95GB)
[5/8] Inlining methods... [***] (2.8s @ 9.26GB)
[6/8] Compiling methods... [******] (39.5s @ 7.39GB)
[7/8] Layouting methods... [****] (14.9s @ 5.84GB)
[8/8] Creating image... [*****] (23.8s @ 5.06GB)
120.43MB (46.23%) for code area: 303,936 compilation units
137.73MB (52.88%) for image heap: 999,594 objects and 57 resources
2.33MB ( 0.89%) for other data
260.49MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
17.00MB io.fabric8.openshift-model-operator-6.10.0.jar 40.09MB byte[] for code metadata
14.62MB svm.jar (Native Image) 24.68MB byte[] for java.lang.String
13.48MB io.fabric8.openshift-model-6.10.0.jar 19.74MB java.lang.Class
12.10MB java.base 17.11MB byte[] for reflection metadata
8.61MB io.fabric8.kubernetes-model-core-6.10.0.jar 9.87MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
4.47MB io.fabric8.openshift-model-monitoring-6.10.0.jar 8.92MB java.lang.String
4.44MB io.fabric8.openshift-model-hive-6.10.0.jar 2.90MB com.oracle.svm.core.hub.DynamicHubCompanion
2.78MB i.f.k.10.0.jar 2.38MB java.lang.Object[]
2.65MB io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar 1.85MB java.lang.String[]
2.59MB io.fabric8.openshift-model-operatorhub-6.10.0.jar 1.48MB byte[] for general heap data
36.07MB for 137 more packages 8.72MB for 3626 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
HEAP: Set max heap for improved and more predictable memory usage.
CPU: Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
22.9s (7.8% of total time) in 95 GCs | Peak RSS: 16.90GB | CPU load: 6.61
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
/Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner (executable)
/Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json
========================================================================================================================
Finished generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' in 4m 55s.
So while it's not completely comparing apples to apples, things do appear significantly worse in 3.7.3, with the openshift model dominating the code areas, as opposed to java.base in 3.2. Types registered for reflection are significantly worse in 3.7.3 as well.
With 3.5.0:
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GRAALVM 23.1 JDK 21.2
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /Users/claprun/.sdkman/candidates/java/21.0.2-graalce/bin/native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=FR -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions --strict-image-heap -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner -jar quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner.jar
[INFO] [io.quarkiverse.operatorsdk.bundle.deployment.BundleProcessor] Generating CSV for 'quarkus-operator-sdk-samples-exposedapp' controller -> /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/bundle/quarkus-operator-sdk-samples-exposedapp/manifests/quarkus-operator-sdk-samples-exposedapp.clusterserviceversion.yaml
[INFO] [io.quarkiverse.operatorsdk.bundle.deployment.BundleProcessor] Generating annotations for 'quarkus-operator-sdk-samples-exposedapp' controller -> /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/bundle/quarkus-operator-sdk-samples-exposedapp/metadata/annotations.yaml
[INFO] [io.quarkiverse.operatorsdk.bundle.deployment.BundleProcessor] Generating bundle for 'quarkus-operator-sdk-samples-exposedapp' controller -> /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/bundle/quarkus-operator-sdk-samples-exposedapp/bundle.Dockerfile
[INFO] [io.quarkiverse.operatorsdk.bundle.deployment.BundleProcessor] Generating Custom Resource Definition for 'quarkus-operator-sdk-samples-exposedapp' controller -> /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/bundle/quarkus-operator-sdk-samples-exposedapp/manifests/exposedapps.halkyon.io-v1.crd.yml
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.netty/netty-transport/reflection-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/8] Initializing... (8.0s @ 0.29GB)
Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
Graal compiler: optimization level: 2, target machine: armv8-a
C compiler: cc (apple, arm64, 15.0.0)
Garbage collector: Serial GC (max heap size: 80% of RAM)
3 user-specific feature(s):
- com.oracle.svm.thirdparty.gson.GsonFeature
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
- io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
------------------------------------------------------------------------------------------------------------------------
4 experimental option(s) unlocked:
- '-H:+AllowFoldMethods' (origin(s): command line)
- '-H:BuildOutputJSONFile' (origin(s): command line)
- '-H:-UseServiceLoaderFeature' (origin(s): command line)
- '-H:ReflectionConfigurationResources' (origin(s): 'META-INF/native-image/io.micrometer/micrometer-core/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/lib/io.micrometer.micrometer-core-1.11.1.jar', 'META-INF/native-image/io.netty/netty-transport/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/lib/io.netty.netty-transport-4.1.100.Final.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
- 13.88GB of memory (21.7% of 64.00GB system memory, determined at start)
- 10 thread(s) (100.0% of 10 available processor(s), determined at start)
[2/8] Performing analysis... [17:00:23,192 WARN [io.net.res.dns.DnsServerAddressStreamProviders] Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'
**********] (25.0s @ 3.12GB)
17,235 reachable types (74.2% of 23,226 total)
33,982 reachable fields (71.3% of 47,683 total)
126,631 reachable methods (67.5% of 187,468 total)
7,120 types, 1,206 fields, and 43,751 methods registered for reflection
61 types, 61 fields, and 55 methods registered for JNI access
5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
[3/8] Building universe... (4.9s @ 4.50GB)
[4/8] Parsing methods... [**] (2.6s @ 3.96GB)
[5/8] Inlining methods... [***] (1.2s @ 3.60GB)
[6/8] Compiling methods... [****] (16.9s @ 2.99GB)
[7/8] Layouting methods... [***] (5.6s @ 4.28GB)
[8/8] Creating image... [***] (9.0s @ 2.15GB)
41.52MB (46.91%) for code area: 93,175 compilation units
46.00MB (51.97%) for image heap: 458,097 objects and 57 resources
1014.56kB ( 1.12%) for other data
88.51MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
12.07MB java.base 13.96MB byte[] for code metadata
4.92MB svm.jar (Native Image) 6.96MB byte[] for java.lang.String
3.17MB io.fabric8.kubernetes-model-core-6.8.1.jar 4.55MB java.lang.Class
1.80MB c.f.jackson.core.jackson-databind-2.15.2.jar 4.16MB java.lang.String
1.39MB q.6.4-SNAPSHOT-runner.jar 3.75MB byte[] for reflection metadata
1.34MB io.fabric8.openshift-model-6.8.1.jar 1.93MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
1.15MB org.bouncycastle.bcprov-jdk18on-1.76.jar 1.45MB com.oracle.svm.core.hub.DynamicHubCompanion
1.11MB modified-io.vertx.vertx-core-4.4.5.jar 1.15MB byte[] for general heap data
630.71kB io.fabric8.openshift-model-operator-6.8.1.jar 865.65kB java.lang.String[]
561.52kB com.fasterxml.jackson.core.jackson-core-2.15.2.jar 765.25kB java.lang.Object[]
12.98MB for 137 more packages 6.51MB for 3606 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
HEAP: Set max heap for improved and more predictable memory usage.
CPU: Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
5.9s (7.9% of total time) in 62 GCs | Peak RSS: 7.52GB | CPU load: 5.32
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
/Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner (executable)
/Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' in 1m 14s.
With 3.6.0:
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GRAALVM 23.1 JDK 21.0.2+13-jvmci-23.1-b30
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /Users/claprun/.sdkman/candidates/java/21.0.2-graalce/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dlogging.initial-configurator.min-level=500 -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Duser.language=en -J-Duser.country=FR -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions --strict-image-heap -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner -jar quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner.jar
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.netty/netty-transport/reflection-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/8] Initializing... (9.1s @ 0.33GB)
Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
Graal compiler: optimization level: 2, target machine: armv8-a
C compiler: cc (apple, arm64, 15.0.0)
Garbage collector: Serial GC (max heap size: 80% of RAM)
3 user-specific feature(s):
- com.oracle.svm.thirdparty.gson.GsonFeature
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
- io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
------------------------------------------------------------------------------------------------------------------------
4 experimental option(s) unlocked:
- '-H:+AllowFoldMethods' (origin(s): command line)
- '-H:BuildOutputJSONFile' (origin(s): command line)
- '-H:-UseServiceLoaderFeature' (origin(s): command line)
- '-H:ReflectionConfigurationResources' (origin(s): 'META-INF/native-image/io.micrometer/micrometer-core/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/lib/io.micrometer.micrometer-core-1.11.5.jar', 'META-INF/native-image/io.netty/netty-transport/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/lib/io.netty.netty-transport-4.1.100.Final.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
- 11.40GB of memory (17.8% of 64.00GB system memory, determined at start)
- 10 thread(s) (100.0% of 10 available processor(s), determined at start)
[2/8] Performing analysis... [17:08:25,416 WARN [io.net.res.dns.DnsServerAddressStreamProviders] Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'
*****] (156.8s @ 6.09GB)
34,253 reachable types (94.7% of 36,152 total)
72,364 reachable fields (84.1% of 86,072 total)
336,541 reachable methods (86.5% of 389,146 total)
24,179 types, 51,594 fields, and 215,026 methods registered for reflection
61 types, 61 fields, and 55 methods registered for JNI access
5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
[3/8] Building universe... (10.9s @ 6.70GB)
[4/8] Parsing methods... [**] (4.5s @ 8.24GB)
[5/8] Inlining methods... [***] (2.2s @ 7.43GB)
[6/8] Compiling methods... [******] (37.5s @ 4.05GB)
[7/8] Layouting methods... [****] (13.6s @ 6.02GB)
[8/8] Creating image... [*****] (21.7s @ 7.56GB)
119.17MB (46.16%) for code area: 301,358 compilation units
136.66MB (52.94%) for image heap: 992,513 objects and 57 resources
2.32MB ( 0.90%) for other data
258.14MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
17.00MB io.fabric8.openshift-model-operator-6.9.2.jar 39.68MB byte[] for code metadata
14.52MB svm.jar (Native Image) 24.52MB byte[] for java.lang.String
13.48MB io.fabric8.openshift-model-6.9.2.jar 19.63MB java.lang.Class
12.08MB java.base 16.97MB byte[] for reflection metadata
8.49MB io.fabric8.kubernetes-model-core-6.9.2.jar 9.80MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
4.47MB io.fabric8.openshift-model-monitoring-6.9.2.jar 8.85MB java.lang.String
4.44MB io.fabric8.openshift-model-hive-6.9.2.jar 2.87MB com.oracle.svm.core.hub.DynamicHubCompanion
2.78MB i.f.k.9.2.jar 2.37MB java.lang.Object[]
2.59MB io.fabric8.openshift-model-operatorhub-6.9.2.jar 1.84MB java.lang.String[]
2.32MB io.fabric8.kubernetes-model-flowcontrol-6.9.2.jar 1.47MB byte[] for general heap data
35.39MB for 137 more packages 8.65MB for 3624 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
HEAP: Set max heap for improved and more predictable memory usage.
CPU: Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
24.7s (9.6% of total time) in 126 GCs | Peak RSS: 12.17GB | CPU load: 6.19
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
/Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner (executable)
/Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' in 4m 17s.
Thanks for the info Chris. Indeed the operator presents the same symptoms as seen in Vincent's application. We'll look into. To replicate:
git clone https://github.com/quarkiverse/quarkus-operator-sdk
cd quarkus-operator-sdk
mvn install -DskipTests
cd samples/exposedapp
mvn package -DskipTests -Dnative -Dnative.surefire.skip
p.s. @metacosm Why doesn't the sample operator app honor -DskipTests? I had to run with mvn debug to discover that -skipTests can only be overriden with -Dnative.surefire.skip?
p.s. @metacosm Why doesn't the sample operator app honor
-DskipTests? I had to run with mvn debug to discover that -skipTestscan only be overriden with-Dnative.surefire.skip?
That looks like a bug, will look into it
More information, I ran native builds with reports and here's the difference of packages between 3.5 and 3.7:
--- reports-3.5/used_packages_quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner_20240215_095352.txt 2024-02-15 09:53:52
+++ reports-3.7/used_packages_quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner_20240215_102428.txt 2024-02-15 10:24:28
@@ -113,6 +113,8 @@
com.sun.management
com.sun.management.internal
com.sun.naming.internal
+com.sun.org.apache.xerces.internal.jaxp.datatype
+com.sun.org.apache.xerces.internal.util
com.sun.proxy.jdk.proxy1
com.sun.proxy.jdk.proxy2
com.sun.proxy.jdk.proxy3
@@ -149,10 +151,11 @@
io.fabric8.kubernetes.api.model.events.v1
io.fabric8.kubernetes.api.model.events.v1beta1
io.fabric8.kubernetes.api.model.extensions
-io.fabric8.kubernetes.api.model.flowcontrol.v1alpha1
+io.fabric8.kubernetes.api.model.flowcontrol.v1
io.fabric8.kubernetes.api.model.flowcontrol.v1beta1
io.fabric8.kubernetes.api.model.flowcontrol.v1beta2
io.fabric8.kubernetes.api.model.flowcontrol.v1beta3
+io.fabric8.kubernetes.api.model.gatewayapi.v1
io.fabric8.kubernetes.api.model.gatewayapi.v1alpha2
io.fabric8.kubernetes.api.model.gatewayapi.v1beta1
io.fabric8.kubernetes.api.model.internal.apiserver.v1alpha1
@@ -171,6 +174,7 @@
io.fabric8.kubernetes.api.model.scheduling.v1
io.fabric8.kubernetes.api.model.scheduling.v1beta1
io.fabric8.kubernetes.api.model.storage
+io.fabric8.kubernetes.api.model.storage.v1alpha1
io.fabric8.kubernetes.api.model.storage.v1beta1
io.fabric8.kubernetes.api.model.version
io.fabric8.kubernetes.client
@@ -207,11 +211,14 @@
io.fabric8.openshift.api.model.console.v1
io.fabric8.openshift.api.model.console.v1alpha1
io.fabric8.openshift.api.model.hive.agent.v1
+io.fabric8.openshift.api.model.hive.alibabacloud.v1
io.fabric8.openshift.api.model.hive.aws.v1
io.fabric8.openshift.api.model.hive.azure.v1
io.fabric8.openshift.api.model.hive.baremetal.v1
io.fabric8.openshift.api.model.hive.gcp.v1
io.fabric8.openshift.api.model.hive.ibmcloud.v1
+io.fabric8.openshift.api.model.hive.metricsconfig.v1
+io.fabric8.openshift.api.model.hive.none.v1
io.fabric8.openshift.api.model.hive.openstack.v1
io.fabric8.openshift.api.model.hive.ovirt.v1
io.fabric8.openshift.api.model.hive.v1
@@ -239,6 +246,7 @@
io.fabric8.openshift.api.model.miscellaneous.cncf.cni.v1
io.fabric8.openshift.api.model.miscellaneous.imageregistry.operator.v1
io.fabric8.openshift.api.model.miscellaneous.metal3.v1alpha1
+io.fabric8.openshift.api.model.miscellaneous.metal3.v1beta1
io.fabric8.openshift.api.model.miscellaneous.network.operator.v1
io.fabric8.openshift.api.model.monitoring.v1
io.fabric8.openshift.api.model.monitoring.v1alpha1
@@ -325,6 +333,7 @@
io.netty.util.internal
io.netty.util.internal.logging
io.netty.util.internal.shaded.org.jctools.queues
+io.netty.util.internal.shaded.org.jctools.queues.atomic
io.netty.util.internal.shaded.org.jctools.util
io.quarkiverse.operatorsdk.common
io.quarkiverse.operatorsdk.runtime
@@ -368,6 +377,7 @@
io.quarkus.runtime.util
io.quarkus.smallrye.context.runtime
io.quarkus.smallrye.health.runtime
+io.quarkus.vertx
io.quarkus.vertx.core.runtime
io.quarkus.vertx.core.runtime.config
io.quarkus.vertx.core.runtime.context
@@ -387,10 +397,10 @@
io.smallrye.common.classloader
io.smallrye.common.constraint
io.smallrye.common.expression
-io.smallrye.common.function
io.smallrye.common.ref
io.smallrye.common.vertx
io.smallrye.config
+io.smallrye.config._private
io.smallrye.config.common
io.smallrye.config.common.utils
io.smallrye.config.inject
@@ -437,6 +447,7 @@
io.vertx.core.impl.verticle
io.vertx.core.json
io.vertx.core.json.impl
+io.vertx.core.json.jackson
io.vertx.core.logging
io.vertx.core.metrics
io.vertx.core.net
@@ -454,6 +465,7 @@
io.vertx.core.spi.transport
io.vertx.core.streams
io.vertx.core.streams.impl
+io.vertx.core.tracing
io.vertx.ext.web
io.vertx.ext.web.client
io.vertx.ext.web.handler
@@ -523,6 +535,8 @@
javax.security.auth.callback
javax.security.auth.x500
javax.tools
+javax.xml.datatype
+javax.xml.namespace
jdk.internal.access
jdk.internal.event
jdk.internal.icu.impl
@@ -550,6 +564,7 @@
jdk.vm.ci.aarch64
jdk.vm.ci.code
jdk.vm.ci.meta
+jdk.xml.internal
org.HdrHistogram
org.bouncycastle.asn1
org.bouncycastle.asn1.cms
@@ -564,6 +579,7 @@
org.bouncycastle.crypto.constraints
org.bouncycastle.crypto.digests
org.bouncycastle.crypto.engines
+org.bouncycastle.crypto.modes
org.bouncycastle.crypto.params
org.bouncycastle.jcajce.provider.symmetric.util
org.bouncycastle.jcajce.util
@@ -661,6 +677,7 @@
org.snakeyaml.engine.v2.tokens
org.wildfly.common
org.wildfly.common._private
+org.wildfly.common.cpu
org.wildfly.common.lock
org.wildfly.common.math
org.wildfly.common.net
@@ -688,6 +705,7 @@
sun.management
sun.misc
sun.net
+sun.net.dns
sun.net.ext
sun.net.spi
sun.net.util
Looks like for some reason, we're now getting io.quarkus.vertx and XML support, which wasn't previously the case. That said, that doesn't explain why the OpenShift model jars are now so much larger.
Upon further investigation it looks like Graal now looks at Builder classes when it previously wasn't the case so, for example, for the AdmissionReview class, the following methods are now considered by Graal, when they previously were not:
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_0777984050acf10da1493a6170fecab4d6751fcb():AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_0fd11b86103ae5deac94f17cd7ebc9ab0f80db23(AdmissionReview):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_1b146ff3d728cebdeee6ce3b7a16914bc1fa167a(AdmissionReviewFluent, AdmissionReview):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_60c83db85be99307ac1cfa5e02331b73895992f2(AdmissionReviewFluent):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_7d86b4847991d0ea7bc237011e756b6bbf79ade5(AdmissionReview):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_8407036dc50873034984857788b3f548e6463c6d():AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_b76e0fbe8c4fc902f7f35602d471fc03f7a1b049(AdmissionReviewFluent, AdmissionReview):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_d1c32c1d0681c1b2f19acb7ac28dac86d0b61674(AdmissionReviewFluent):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent$ResponseNested_constructor_259ca1fdf585b79b107e52e1c9e93f64803f3dd1(AdmissionReviewFluent, AdmissionResponse):AdmissionReviewFluent$ResponseNested
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent$ResponseNested_constructor_9ede732b5de68aeabe21301372523139b271aeb0(AdmissionReviewFluent, AdmissionResponse):AdmissionReviewFluent$ResponseNested
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent_constructor_8e18a7129d7cfce282cc44631a106586b9f50160():AdmissionReviewFluent
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent_constructor_a3286f0b1cb894d801b61242fb3ee01a918d9ac7():AdmissionReviewFluent
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent_constructor_b36cf7bb0e633398005c903e018b111dd2f934d5(AdmissionReview):AdmissionReviewFluent
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent_constructor_c44ba2db71b9766138f88aac8a2d5f191200375d(AdmissionReview):AdmissionReviewFluent
So with around as many methods added for each type, that certainly explains the size difference. The question, now, is why are the builders considered when they previously weren't.
Thanks @metacosm for your comments. We have replicated the issue now on our side but the amount of data to get through is considerable. The only thing I've been able to observe so far is that the blow out is already present in the Quarkus operator, which makes things easier on our side. We'll reply when we have a good understanding of the causes.
I'm looking into the reason why the builders are now considered. I think that we'll be in good shape if we can figure this out and address this particular issue first.
For some extra insights. I'm working on a demo project to showcase how to work on OpenShift with just the vanilla kubernetes-client-extension: https://github.com/marcnuri-demo/quarkus-kubernetes-client-models/blob/c4123613e94646018d2d4e6ade36267dde056d8a/
Compiling a native image in 3.7.3, produces the following output:
159.55MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
14.53MB io.fabric8.openshift-model-6.10.0.jar 25.02MB byte[] for code metadata
12.94MB java.base 13.04MB byte[] for java.lang.String
9.29MB io.fabric8.kubernetes-model-core-6.10.0.jar 8.99MB byte[] for reflection metadata
8.82MB svm.jar (Native Image) 8.19MB java.lang.Class
2.97MB i.f.k.10.0.jar 5.88MB java.lang.String
2.85MB io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar 5.15MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
2.48MB io.fabric8.kubernetes-model-flowcontrol-6.10.0.jar 1.84MB com.oracle.svm.core.hub.DynamicHubCompanion
1.97MB c.f.jackson.core.jackson-databind-2.16.1.jar 1.36MB java.lang.Object[]
1.76MB io.fabric8.kubernetes-model-autoscaling-6.10.0.jar 1.24MB byte[] for general heap data
1.75MB i.f.kubernetes-model-apiextensions-6.10.0.jar 1.21MB java.lang.String[]
19.24MB for 117 more packages 7.27MB for 3416 more object types
While the same project in 3.5.3:
76.36MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
12.90MB java.base 11.76MB byte[] for code metadata
4.06MB svm.jar (Native Image) 5.93MB byte[] for java.lang.String
3.43MB io.fabric8.kubernetes-model-core-6.8.1.jar 3.95MB java.lang.Class
1.93MB c.f.jackson.core.jackson-databind-2.15.3.jar 3.65MB java.lang.String
1.26MB io.fabric8.openshift-model-6.8.1.jar 2.58MB byte[] for reflection metadata
1.17MB modified-io.vertx.vertx-core-4.4.6.jar 1.27MB com.oracle.svm.core.hub.DynamicHubCompanion
823.45kB q.0.0-SNAPSHOT-runner.jar 1.21MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
582.35kB com.fasterxml.jackson.core.jackson-core-2.15.3.jar 1.12MB byte[] for general heap data
575.69kB org.yaml.snakeyaml-2.1.jar 754.34kB java.lang.String[]
510.70kB org.snakeyaml.snakeyaml-engine-2.6.jar 672.46kB java.lang.Object[]
9.09MB for 117 more packages 6.23MB for 3391 more object types
So the increase in size is also extremely visible here. Since this project is much smaller, maybe it's easier to use it to analyze why now classes such as builders are now being considered.
I've pinpointed the culprit: we added an edit method to all model objects in https://github.com/fabric8io/kubernetes-client/commit/a3efaa3d773de02f42ca16255d66155b8bc97118. Prior to that commit, a native build would report:
more target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/reports/used_methods_quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner_20240215_164459.txt |
grep Builder | wc -l
6398
With that commit, we get:
more target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/reports/used_methods_quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner_20240215_170845.txt |
grep Builder | wc -l
44403
Looking at the changelog and the symptoms, it made sense that this change was the culprit but I wanted to have the numbers to back that claim, first… 😄 Now, the question is how to fix it.
Ok, I guess that commits binds the types to their builders and then combined with:
Collection<ClassInfo> visitableBuilderImpls = fullIndex.getAllKnownImplementors(VISITABLE_BUILDER);
...
populateReflectionRegistrationLists(visitableBuilderImpls, watchedClasses, ignoreJsonDeserialization,
withoutFieldsRegistration,
withFieldsRegistration);
All those builders found are registered for reflection.
Builders create types. By having types create builders that creates a bidirectional dependency which I'm not sure it's really needed. You can still have a builder that can be created out of a type, but done outside of the type and that breaks the cycle. Something for @manusa et al to chew on.
A workaround at the Quarkus level might be to only register those builders for reflection that are considered reachable by the points-to analysis. The concept is explained here and some parts of Quarkus use this, so there are examples in the code base, but I'm not 100% sure it would solve the issue. It would depend on how different the set of reachable builders is vs the set of builder implementations in the jandex index.
great find @metacosm - so the cause is that the read view of class hierarchy via this edit method got connected to all the builders in one fell swoop.
That explains it but definitely very counter to our past efforts of reducing/decoupling the client.
Could a visitor pattern solve this instead?
And yes as @galderz mention only register those that are accessible via the points to analysis could help here !
I was working on providing the resources asked by @zakkak in https://github.com/quarkusio/quarkus/issues/38683#issuecomment-1935486283 I would not be able to make it public (only through a support case). but it feels that you made a lot of progress in the analysis. do you still need something from me?
@vsevel you probably don't need to anymore… though it might be a good idea to keep the work you did around just in case we need it later on! 😉