glassfish-hk2
glassfish-hk2 copied to clipboard
eclipse-ee4j
Race condition when initializing ClassAnalyzer with CdiInjecteeSkippingClassAnalyzer
For now several months, we are getting the following error for some of the builds we execute (I estimate it to 1-5% of all the builds). It uses Jersey 3.0.2, Weld 4.0.2.Final with Java 16. The error occurs when executing a JUnit test initializing CDI & JAX-RS server, similar to this test https://github.com/nhenneaux/jersey-httpclient-connector/blob/master/src/test/java/com/github/nhenneaux/jersey/connector/httpclient/JettyServerTest.java#L222
Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
at org.jvnet.hk2.internal.ServiceLocatorImpl.getAnalyzer(ServiceLocatorImpl.java:2468)
at org.jvnet.hk2.internal.Utilities.getClassAnalyzer(Utilities.java:143)
at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:969)
I suspect a race condition occurs since the class analyzer uses a HashMap and not a thread safe map (java.util.concurrent.ConcurrentMap). Indeed one CPU core cache could see the map full and another the map empty. https://github.com/eclipse-ee4j/glassfish-hk2/blob/master/hk2-locator/src/main/java/org/jvnet/hk2/internal/ServiceLocatorImpl.java#L2464
Full stacktrace
java.lang.IllegalStateException: jakarta.servlet.ServletException: org.glassfish.jersey.servlet.ServletContainer-f7acce1==org.glassfish.jersey.servlet.ServletContainer@e976386e{jsp=null,order=-1,inst=true,async=true,src=EMBEDDED:null,STARTED}
...
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: jakarta.servlet.ServletException: org.glassfish.jersey.servlet.ServletContainer-f7acce1==org.glassfish.jersey.servlet.ServletContainer@e976386e{jsp=null,order=-1,inst=true,async=true,src=EMBEDDED:null,STARTED}
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:651)
at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:415)
at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$2(ServletHandler.java:690)
at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:714)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:891)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
at org.eclipse.jetty.server.handler.StatisticsHandler.doStart(StatisticsHandler.java:242)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
at org.eclipse.jetty.server.Server.start(Server.java:469)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
at org.eclipse.jetty.server.Server.doStart(Server.java:414)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
... ... 9 more
Caused by: A MultiException has 1 exceptions. They are:
1. java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
at org.jvnet.hk2.internal.Collector.throwIfErrors(Collector.java:65)
at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:970)
at org.jvnet.hk2.internal.ServiceLocatorImpl.inject(ServiceLocatorImpl.java:1014)
at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.inject(AbstractHk2InjectionManager.java:207)
at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.inject(ImmediateHk2InjectionManager.java:30)
at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:769)
at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:161)
at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64)
at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:694)
at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:717)
at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:64)
at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:87)
at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:129)
at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:72)
at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:112)
at org.glassfish.jersey.ext.cdi1x.internal.GenericCdiBeanSupplier.get(GenericCdiBeanSupplier.java:42)
at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:53)
at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:129)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466)
at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:46)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2103)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.lambda$getAllServiceHolders$0(AbstractHk2InjectionManager.java:136)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1242)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getAllServiceHolders(AbstractHk2InjectionManager.java:140)
at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getAllServiceHolders(ImmediateHk2InjectionManager.java:30)
at org.glassfish.jersey.internal.inject.Providers.getServiceHolders(Providers.java:307)
at org.glassfish.jersey.internal.inject.Providers.getCustomProviders(Providers.java:151)
at org.glassfish.jersey.server.internal.inject.ParamExtractorConfigurator.lambda$init$0(ParamExtractorConfigurator.java:49)
at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:317)
at org.glassfish.jersey.server.internal.inject.MultivaluedParameterExtractorFactory.get(MultivaluedParameterExtractorFactory.java:68)
at org.glassfish.jersey.server.internal.inject.AbstractValueParamProvider.get(AbstractValueParamProvider.java:67)
at org.glassfish.jersey.server.internal.inject.PathParamValueParamProvider.createValueProvider(PathParamValueParamProvider.java:71)
at org.glassfish.jersey.server.internal.inject.AbstractValueParamProvider.getValueProvider(AbstractValueParamProvider.java:93)
at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParamValueProvider(ParameterValueHelper.java:148)
at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.createValueProviders(ParameterValueHelper.java:110)
at org.glassfish.jersey.server.model.ResourceMethodValidator.checkValueProviders(ResourceMethodValidator.java:172)
at org.glassfish.jersey.server.model.ResourceMethodValidator.checkMethod(ResourceMethodValidator.java:81)
at org.glassfish.jersey.server.model.ResourceMethodValidator.visitJaxrsResourceMethod(ResourceMethodValidator.java:77)
at org.glassfish.jersey.server.model.ResourceMethodValidator.visitResourceMethod(ResourceMethodValidator.java:67)
at org.glassfish.jersey.server.model.ResourceMethod.accept(ResourceMethod.java:876)
at org.glassfish.jersey.server.model.ComponentModelValidator.validateWithErrors(ComponentModelValidator.java:126)
at org.glassfish.jersey.server.model.ComponentModelValidator.validateWithErrors(ComponentModelValidator.java:132)
at org.glassfish.jersey.server.model.ComponentModelValidator.validateWithErrors(ComponentModelValidator.java:132)
at org.glassfish.jersey.server.model.ComponentModelValidator.validateWithErrors(ComponentModelValidator.java:132)
at org.glassfish.jersey.server.model.ComponentModelValidator.access$000(ComponentModelValidator.java:55)
at org.glassfish.jersey.server.model.ComponentModelValidator$1.run(ComponentModelValidator.java:116)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.server.model.ComponentModelValidator.validate(ComponentModelValidator.java:111)
at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:366)
at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$1(ApplicationHandler.java:293)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:232)
at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:292)
at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:259)
at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:311)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:347)
at jakarta.servlet.GenericServlet.init(GenericServlet.java:178)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:633)
... 38 more
Caused by: java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
at org.jvnet.hk2.internal.ServiceLocatorImpl.getAnalyzer(ServiceLocatorImpl.java:2468)
at org.jvnet.hk2.internal.Utilities.getClassAnalyzer(Utilities.java:143)
at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:969)
... 110 more
It seems it affected much more users, year after year, but I didn't find any explanation except "something uses the container before it finished it's initialization" until now. I believe this is not the issue of HK2, but of the project using Jersey with HK2 where is something wrong.
The example project always passed all tests on my computer with JDK17 (except log messages from JaCoCo instrumentation; it passed also in eclipse without these messages).
But what finally attracted my eye is the class org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider which is in jersey-cdi1x project, while the example uses jersey-cdi2se. I would simply kick the cdi1x out of dependencies (is it doable?); I guess there may be some issue with your dependencies which causes that the container initializes services in random order (as the order is not defined), but I am not sure with it. I would really need some reproducer for that including all your dependencies.
However I am still convinced that using ConcurrentHashMap cannot resolve the problem, because all accesses to it are synchronized.
Simply said - it tries to inject something bound to the CdiInjecteeSkippingClassAnalyzer while CdiComponentProvider.initialize wan't called yet, did not finish or the beanManager was null.
The CdiServerComponentProvider is an interesting class too - no synchronization. And the ComponentProvider.initialize has suspicious javadoc saying
Initializes the component provider with a reference to a injection manager instance, which will get used in the application to manage individual components. Providers should keep a reference to the injection manager for later use. This method will be invoked prior to any bind method calls. The injection manager parameter will not be fully initialized at the time of invocation and should be used as a reference only.
But then the CdiComponentProvider.bindHk2ClassAnalyzer breaks this rule, because it uses it before the initialize method ended. Probably this is the real cause of the issue ...? Then the problem would be in Jersey, not in HK2. But again, I would simply run away from CDI 1.1 now.
Thanks for your follow-up!!
I am using Weld 4.0.2 with org.glassfish.jersey.ext.cdi::jersey-weld2-se which is under the hood depends on jersey-cdi1x. It is compatible with Weld 4.
I agree it probably comes with the order of initialization which is not deterministic. However it might also come with a concurrency issue with some value in cpu cache not matching the main memory.
I already tried to replace hk2 with pure cdi injection but it requires major rework in the way we use them now.
Here is all the dependencies
| | +- org.jboss.weld.se:weld-se-core:jar:4.0.2.Final:compile
| | | +- org.jboss.weld.environment:weld-environment-common:jar:4.0.2.Final:compile
| | | +- org.jboss.weld.probe:weld-probe-core:jar:4.0.2.Final:compile
| | | \- org.jboss.classfilewriter:jboss-classfilewriter:jar:1.2.5.Final:compile
| | +- org.jboss.weld:weld-core-impl:jar:4.0.2.Final:compile
| | | \- org.jboss.weld:weld-spi:jar:4.0.SP1:compile
| | +- org.jboss.weld:weld-api:jar:4.0.SP1:compile
| | \- jakarta.servlet:jakarta.servlet-api:jar:5.0.0:compile
| \- jakarta.activation:jakarta.activation-api:jar:2.0.1:compile
| +- jakarta.inject:jakarta.inject-api:jar:2.0.0:compile
| +- jakarta.enterprise:jakarta.enterprise.cdi-api:jar:3.0.0:compile
| | \- jakarta.interceptor:jakarta.interceptor-api:jar:2.0.0:compile
| +- jakarta.annotation:jakarta.annotation-api:jar:2.0.0:compile
| \- org.slf4j:slf4j-api:jar:1.7.32:compile
| | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:3.0.1:compile
| | +- com.sun.xml.bind:jaxb-impl:jar:3.0.2:compile
| | | \- com.sun.xml.bind:jaxb-core:jar:3.0.2:compile
| | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.12.5:compile
| | \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.12.5:compile
| +- io.swagger:swagger-annotations:jar:1.6.2:compile
| +- org.glassfish.jersey.media:jersey-media-json-jackson:jar:3.0.3:compile
| | +- org.glassfish.jersey.core:jersey-common:jar:3.0.3:compile
| | +- org.glassfish.jersey.ext:jersey-entity-filtering:jar:3.0.3:compile
| | +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.5:compile
| | \- javax.xml.bind:jaxb-api:jar:2.3.1:compile
| \- jakarta.ws.rs:jakarta.ws.rs-api:jar:3.0.0:compile
| | \- org.ehcache:ehcache:jar:3.9.3:compile
| +- org.glassfish.jersey.ext.cdi:jersey-weld2-se:jar:3.0.3:compile
| | +- org.glassfish.jersey.core:jersey-server:jar:3.0.3:compile
| | | \- org.glassfish.jersey.core:jersey-client:jar:3.0.3:compile
| | \- org.glassfish.jersey.ext.cdi:jersey-cdi1x:jar:3.0.3:compile
| +- org.glassfish.jersey.inject:jersey-hk2:jar:3.0.3:compile
| | +- org.glassfish.hk2:hk2-locator:jar:3.0.1:compile
| | | +- org.glassfish.hk2.external:aopalliance-repackaged:jar:3.0.1:compile
| | | +- org.glassfish.hk2:hk2-api:jar:3.0.1:compile
| | | \- org.glassfish.hk2:hk2-utils:jar:3.0.1:compile
| | \- org.javassist:javassist:jar:3.25.0-GA:compile
| +- org.eclipse.jetty:jetty-servlet:jar:11.0.7:compile
| | \- org.eclipse.jetty:jetty-security:jar:11.0.7:compile
| +- org.glassfish.jersey.containers:jersey-container-jetty-servlet:jar:3.0.3:compile
| | +- org.glassfish.jersey.containers:jersey-container-servlet:jar:3.0.3:compile
| | +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:3.0.3:compile
| | \- org.glassfish.jersey.containers:jersey-container-jetty-http:jar:3.0.3:compile
| +- org.glassfish.jersey.ext:jersey-proxy-client:jar:3.0.3:compile
| +- org.glassfish.jersey.ext:jersey-bean-validation:jar:3.0.3:compile
| +- org.hibernate.validator:hibernate-validator:jar:7.0.1.Final:compile
| | +- org.jboss.logging:jboss-logging:jar:3.4.1.Final:compile
| | \- com.fasterxml:classmate:jar:1.5.1:compile
| +- org.glassfish.jersey.media:jersey-media-multipart:jar:3.0.3:compile
| | \- org.jvnet.mimepull:mimepull:jar:1.9.13:compile
| +- org.eclipse.jetty.http2:http2-server:jar:11.0.7:compile
| | +- org.eclipse.jetty.http2:http2-common:jar:11.0.7:compile
| | | \- org.eclipse.jetty.http2:http2-hpack:jar:11.0.7:compile
| | \- org.eclipse.jetty:jetty-server:jar:11.0.7:compile
| | \- org.eclipse.jetty:jetty-http:jar:11.0.7:compile
| +- org.eclipse.jetty:jetty-alpn-java-server:jar:11.0.7:compile
| | +- org.eclipse.jetty:jetty-io:jar:11.0.7:compile
| | | \- org.eclipse.jetty:jetty-util:jar:11.0.7:compile
| | \- org.eclipse.jetty:jetty-alpn-server:jar:11.0.7:compile
| +- com.github.nhenneaux.jersey.connector.httpclient:jersey-httpclient-connector:jar:1.0.10:compile
| \- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.5:compile
| \- com.github.stefanbirkner:system-lambda:jar:1.1.0:compile
+- org.mockito:mockito-core:jar:3.12.4:compile
| +- net.bytebuddy:byte-buddy:jar:1.11.13:compile
| +- net.bytebuddy:byte-buddy-agent:jar:1.11.13:compile
| \- org.objenesis:objenesis:jar:3.2:compile
+- org.slf4j:slf4j-simple:jar:1.7.32:test
+- jakarta.validation:jakarta.validation-api:jar:3.0.0:compile
+- com.fasterxml.jackson.core:jackson-databind:jar:2.12.5:compile
| \- com.fasterxml.jackson.core:jackson-core:jar:2.12.5:compile
| +- org.reflections:reflections:jar:0.9.11:test
| | \- com.google.guava:guava:jar:30.1-jre:test
| | +- com.google.guava:failureaccess:jar:1.0.1:test
| | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:test
| | +- com.google.code.findbugs:jsr305:jar:3.0.2:test
| | +- org.checkerframework:checker-qual:jar:3.5.0:test
| | +- com.google.errorprone:error_prone_annotations:jar:2.3.4:test
| | \- com.google.j2objc:j2objc-annotations:jar:1.3:test
| \- org.apache.commons:commons-lang3:jar:3.11:test
+- jakarta.el:jakarta.el-api:jar:4.0.0:compile
+- org.glassfish:jakarta.el:jar:4.0.2:compile
+- org.junit.jupiter:junit-jupiter-engine:jar:5.8.1:test
| +- org.junit.platform:junit-platform-engine:jar:1.8.1:test
| | +- org.opentest4j:opentest4j:jar:1.2.0:test
| | \- org.junit.platform:junit-platform-commons:jar:1.8.1:test
| +- org.junit.jupiter:junit-jupiter-api:jar:5.8.1:test
| \- org.apiguardian:apiguardian-api:jar:1.1.2:test
+- org.awaitility:awaitility:jar:3.1.6:test
| +- org.hamcrest:hamcrest-library:jar:1.3:test
| \- org.hamcrest:hamcrest-core:jar:1.3:test
+- org.hamcrest:hamcrest:jar:2.2:test
\- org.mockito:mockito-junit-jupiter:jar:3.12.4:test
Hmm, the referred example uses jersey-cdi2-se instead of jersey-weld2-se ... btw ... HK2 is nearly a concurrent of Weld, so that really can cause some issues ... it should use just one of them.
Also I see that in service files of cdi1x project is the CdiComponentProvider twice, once as an extension and once as a ComponentProvider impl - that can probably cause redundant initialization too. Or two standalone instances of the same class used as two standalone services ...
If cpu caches would work like this, nothing would work ever ... prove it (the map is still protected by synchronized methods).
Btw I was digging in my old examples and I found this:
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-server</artifactId>
<scope>provided</scope>
</dependency>
<!-- If we skip this dependency, jersey-client throws java.lang.IllegalStateException: InjectionManagerFactory not found. -->
<!-- With that we must have META-INF/beans.xml in src/test/resources -->
<dependency>
<groupId>org.glassfish.jersey.inject</groupId>
<artifactId>jersey-cdi2-se</artifactId>
</dependency>
<dependency>
<groupId>org.jboss.weld.se</groupId>
<artifactId>weld-se-core</artifactId>
</dependency>
I'm also getting this error while the applications runs in production. It occurred the 11/01/2022 for a service started the 20/12/2021. I include a truncated stack of the error.
jakarta.ws.rs.ProcessingException: A MultiException has 1 exceptions. They are:
1. java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:309) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[app.jar:x.x.x]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[app.jar:x.x.x]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[app.jar:x.x.x]
at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[app.jar:x.x.x]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:461) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.proxy.WebResourceFactory.invoke(WebResourceFactory.java:329) ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at jdk.proxy2.$Proxy134.partitionKeyCaptureBatchAcquireLock(Unknown Source) ~[?:?]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class [app.jar:x.x.x]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.glassfish.hk2.api.MultiException: A MultiException has 1 exceptions. They are:
1. java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
at org.jvnet.hk2.internal.Collector.throwIfErrors(Collector.java:65) ~[app.jar:x.x.x]
at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:970) ~[app.jar:x.x.x]
at org.jvnet.hk2.internal.ServiceLocatorImpl.inject(ServiceLocatorImpl.java:1014) ~[app.jar:x.x.x]
at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.inject(AbstractHk2InjectionManager.java:212) ~[app.jar:x.x.x]
at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.inject(ImmediateHk2InjectionManager.java:30) ~[app.jar:x.x.x]
at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:776) ~[app.jar:x.x.x]
at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:161) ~[app.jar:x.x.x]
at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64) ~[app.jar:x.x.x]
at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) ~[app.jar:x.x.x]
at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) ~[app.jar:x.x.x]
at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:694) ~[app.jar:x.x.x]
at org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:252) ~[app.jar:x.x.x]
at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:114) ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at jdk.proxy2.$Proxy177.signature(Unknown Source) ~[?:?]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at org.glassfish.jersey.client.ClientFilteringStages$RequestFilteringStage.apply(ClientFilteringStages.java:144) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.ClientFilteringStages$RequestFilteringStage.apply(ClientFilteringStages.java:132) ~[app.jar:x.x.x]
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[app.jar:x.x.x]
... 36 more
Caused by: java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
at org.jvnet.hk2.internal.ServiceLocatorImpl.getAnalyzer(ServiceLocatorImpl.java:2468) ~[app.jar:x.x.x]
at org.jvnet.hk2.internal.Utilities.getClassAnalyzer(Utilities.java:143) ~[app.jar:x.x.x]
at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:969) ~[app.jar:x.x.x]
at org.jvnet.hk2.internal.ServiceLocatorImpl.inject(ServiceLocatorImpl.java:1014) ~[app.jar:x.x.x]
at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.inject(AbstractHk2InjectionManager.java:212) ~[app.jar:x.x.x]
at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.inject(ImmediateHk2InjectionManager.java:30) ~[app.jar:x.x.x]
at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:776) ~[app.jar:x.x.x]
at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:161) ~[app.jar:x.x.x]
at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64) ~[app.jar:x.x.x]
at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) ~[app.jar:x.x.x]
at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) ~[app.jar:x.x.x]
at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:694) ~[app.jar:x.x.x]
at org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:252) ~[app.jar:x.x.x]
at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:114) ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at jdk.proxy2.$Proxy177.signature(Unknown Source) ~[?:?]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at internal.class ~[app.jar:x.x.x]
at org.glassfish.jersey.client.ClientFilteringStages$RequestFilteringStage.apply(ClientFilteringStages.java:144) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.ClientFilteringStages$RequestFilteringStage.apply(ClientFilteringStages.java:132) ~[app.jar:x.x.x]
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147) ~[app.jar:x.x.x]
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[app.jar:x.x.x]
... 36 more
@dmatej I think I found where the problem lie. It occurs when there is a concurrent initialization from Jersey server but also from each new client. Each new client triggers the initialization of injection which makes things worse.
Hi @dmatej Were you able to dig further on this issue? We are still impacted by it from time to time. Especially in CI but also for some use case in live code. I'm not sure the associated MR would help but I think it provides an extra guarantee about the consistency of the store in case of race condition.
Hi, We're getting this error quite suddenly now, first occurrence was May 11th, before this we never saw this error. It's our production environment, we've had no new deployments since mid-April.
Any suggestions how to deal with this?
Hi, We're getting this error quite suddenly now, first occurrence was May 11th, before this we never saw this error. It's our production environment, we've had no new deployments since mid-April.
Any suggestions how to deal with this?
With the same version of Jetty, HK2 and Jersey? Could you create a reproducer?
Hi @dmatej,
I'm afraid thats not possible: the error does not occur for every request, but maybe 4 or 5 times a day, with over a hundred requests per day.
We are using Payara Server Enterprise 5.49.0 #badassfish (build 839).
The issue is still happening on a regular basis in CI and production with Jersey 3.1.1 & Jetty 11.0.15.
It happens more when the Jersey client is instantiated concurrently. What about merging MR https://github.com/eclipse-ee4j/glassfish-hk2/pull/560 since it is quite harmless and might fix the issue?
I don't remember so. It happens when doing concurrent client instantiations and I don't on this project. https://github.com/nhenneaux/jersey-httpclient-connector/blob/master/src/test/java/com/github/nhenneaux/jersey/connector/httpclient/JettyServerTest.java#L412-L413