quarkus-langchain4j
quarkus-langchain4j copied to clipboard
Published
20 hours ago •
quarkiverse
quarkiverse
Integration tests fail non-deterministically in native mode
Issue keeps popping up in https://github.com/quarkiverse/quarkiverse/issues/195
Originally seen and discussed in https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/GraalVM.20-.20freezing
The issue seems related (although not confirmed yet) to the order in which classes are getting initialized (we have observed a similar issue in https://github.com/quarkusio/quarkus/issues/37657).
The failures seem related to the initialization of classes in org.apache.xmlbeans, org.w3.x2000.x09.xmldsig, org.openxmlformats and possibly more xml related packages.
Trying to reproduce locally I was able to get another interesting error as well:
[1/8] Initializing... (3.3s @ 0.27GB)
Java version: 21.0.2+13-LTS, vendor version: Mandrel-23.1.2.0-Final
Graal compiler: optimization level: 2, target machine: x86-64-v3
C compiler: gcc (redhat, x86_64, 8.5.0)
Garbage collector: Serial GC (max heap size: 80% of RAM)
5 user-specific feature(s):
- com.oracle.svm.thirdparty.gson.GsonFeature
- io.quarkiverse.poi.runtime.graal.POIFeature: Quarkus runtime initialization for Apache POI
- io.quarkus.awt.runtime.graal.DarwinAwtFeature
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
- io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
------------------------------------------------------------------------------------------------------------------------
3 experimental option(s) unlocked:
- '-H:+AllowFoldMethods' (origin(s): command line)
- '-H:BuildOutputJSONFile' (origin(s): command line)
- '-H:-UseServiceLoaderFeature' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
- 7.75GB of memory (12.5% of 61.94GB system memory, determined at start)
- 32 thread(s) (100.0% of 32 available processor(s), determined at start)
2024-02-26T12:59:25.425180037Z main ERROR Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
[2/8] Performing analysis... [] (0.9s @ 0.91GB)
8,390 reachable types (81.4% of 10,307 total)
5,101 reachable fields (68.3% of 7,467 total)
3,433 reachable methods (11.8% of 28,983 total)
4,462 types, 4,591 fields, and 6,744 methods registered for reflection
Fatal error: com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing io.quarkiverse.langchain4j.QuarkusInMemoryEmbeddingJsonCodecFactory$Codec.fromJson(QuarkusInMemoryEmbeddingJsonCodecFactory.java:25)
Parsing context:
at dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore.fromJson(InMemoryEmbeddingStore.java:140)
at dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore.fromFile(InMemoryEmbeddingStore.java:146)
at static root method.(Unknown Source)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisError.parsingError(AnalysisError.java:149)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:184)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureFlowsGraphCreated(MethodTypeFlow.java:153)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.getOrCreateMethodFlowsGraphInfo(MethodTypeFlow.java:111)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultVirtualInvokeTypeFlow.onObservedUpdate(DefaultVirtualInvokeTypeFlow.java:114)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:620)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.PointsToAnalysis$1.run(PointsToAnalysis.java:491)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:187)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:171)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: org.graalvm.compiler.java.BytecodeParser$BytecodeParserError: org.graalvm.compiler.debug.GraalError: com.oracle.svm.core.util.UserError$UserException: Class initialization of io.quarkiverse.langchain4j.QuarkusJsonCodecFactory$ObjectMapperHolder failed. Use the option
'--initialize-at-run-time=io.quarkiverse.langchain4j.QuarkusJsonCodecFactory$ObjectMapperHolder'
to explicitly request initialization of this class at run time.
at parsing io.quarkiverse.langchain4j.QuarkusInMemoryEmbeddingJsonCodecFactory$Codec.fromJson(QuarkusInMemoryEmbeddingJsonCodecFactory.java:25)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.throwParserError(BytecodeParser.java:2558)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.throwParserError(SharedGraphBuilderPhase.java:182)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3439)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.iterateBytecodesForBlock(SharedGraphBuilderPhase.java:743)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.handleBytecodeBlock(BytecodeParser.java:3391)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:3233)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1137)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.build(SharedGraphBuilderPhase.java:162)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:1029)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:101)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase.run(SharedGraphBuilderPhase.java:116)
at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:434)
at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.AnalysisParsedGraph.parseBytecode(AnalysisParsedGraph.java:146)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.parseGraph(AnalysisMethod.java:895)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsedHelper(AnalysisMethod.java:860)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsed(AnalysisMethod.java:843)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:186)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:621)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:167)
... 13 more
Caused by: org.graalvm.compiler.debug.GraalError: com.oracle.svm.core.util.UserError$UserException: Class initialization of io.quarkiverse.langchain4j.QuarkusJsonCodecFactory$ObjectMapperHolder failed. Use the option
'--initialize-at-run-time=io.quarkiverse.langchain4j.QuarkusJsonCodecFactory$ObjectMapperHolder'
to explicitly request initialization of this class at run time.
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisFuture.setException(AnalysisFuture.java:49)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:322)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisFuture.ensureDone(AnalysisFuture.java:63)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.ensureOnTypeReachableTaskDone(AnalysisType.java:696)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.onReachable(AnalysisType.java:590)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AtomicUtils.atomicSetAndRun(AtomicUtils.java:49)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.lambda$registerAsReachable$8(AnalysisType.java:562)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.forAllSuperTypes(AnalysisType.java:676)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.forAllSuperTypes(AnalysisType.java:659)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.forAllSuperTypes(AnalysisType.java:655)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.registerAsReachable(AnalysisType.java:562)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisUniverse.lookupAllowUnresolved(AnalysisUniverse.java:368)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.infrastructure.AnalysisConstantPool.lookupField(AnalysisConstantPool.java:42)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.lookupField(BytecodeParser.java:4364)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genGetStatic(BytecodeParser.java:4924)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5435)
at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3431)
... 32 more
Caused by: com.oracle.svm.core.util.UserError$UserException: Class initialization of io.quarkiverse.langchain4j.QuarkusJsonCodecFactory$ObjectMapperHolder failed. Use the option
'--initialize-at-run-time=io.quarkiverse.langchain4j.QuarkusJsonCodecFactory$ObjectMapperHolder'
to explicitly request initialization of this class at run time.
at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:85)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.ensureClassInitialized(ClassInitializationSupport.java:195)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.AllowAllHostedUsagesClassInitializationSupport.computeInitKindAndMaybeInitializeClass(AllowAllHostedUsagesClassInitializationSupport.java:191)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.AllowAllHostedUsagesClassInitializationSupport.computeInitKindAndMaybeInitializeClass(AllowAllHostedUsagesClassInitializationSupport.java:129)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.maybeInitializeAtBuildTime(ClassInitializationSupport.java:161)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.maybeInitializeAtBuildTime(ClassInitializationSupport.java:150)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.SVMHost.onTypeReachable(SVMHost.java:310)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisUniverse.onTypeReachable(AnalysisUniverse.java:699)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.lambda$new$0(AnalysisType.java:310)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
... 47 more
Caused by: java.lang.ExceptionInInitializerError
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.ensureClassInitialized(ClassInitializationSupport.java:177)
... 56 more
Caused by: java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.instance(java.lang.Class, java.lang.annotation.Annotation[])" because the return value of "io.quarkus.arc.Arc.container()" is null
at io.quarkiverse.langchain4j.QuarkusJsonCodecFactory$ObjectMapperHolder.<clinit>(QuarkusJsonCodecFactory.java:70)
... 59 more
Trying to reproduce locally I was able to get another interesting error as well:
What command did you execute and got that error? Are you perhaps using a newer version of GraalVM?
What command did you execute and got that error?
mvn clean install -Dnative -Dquarkus.native.container-build=true --fail-at-end -pl integration-tests/openai
Are you perhaps using a newer version of GraalVM?
No that was using tag jdk-21 of quay.io/quarkus/ubi-quarkus-mandrel-builder-image. A difference that might be important is that I used Quarkus built from main instead of 3.7.3
Note this is not deterministic as well...
I am uploading for future reference the logs of the failed jobs on the github runners:
Looking at the 3 logs it appears that the issue is related to the Class initialization of org.apache.xmlbeans.metadata.system.sXMLLANG.TypeSystemHolder in parallel with org.apache.xmlbeans.metadata.system.sXMLSCHEMA.TypeSystemHolder (and I suspect any of the TypeSystemHolders under org.apache.xmlbeans.metadata.system.*) .
Interestingly the initialization of org.apache.xmlbeans.metadata.system.sXMLLANG.TypeSystemHolder seems to be consistently triggered by the class initialization of org.openxmlformats.schemas.drawingml.x2006.chart.STLayoutModel while the initialization of org.apache.xmlbeans.metadata.system.sXMLSCHEMA.TypeSystemHolder is triggered by a different class in each case.
Inspecting the class initializers they seem to invoke the same constructor which then ends up calling org.apache.xmlbeans.impl.schema.SchemaTypeLoaderImpl.build which in turn invokes java.lang.Class.forName0 which I suspect results in some kind of deadlock when invoked for the same class.
Still trying to figure out the exact issue for a proper fix.
The trivial work around I can think of is to register these classes for runtime initialization.
@geoand I finally put some time aside to have a look at this again but it looks like it's no longer reproducing. Is that right? Have you noticed any instances of it?