quarkus
quarkus copied to clipboard
quarkusio
OpenTelemetry and smallrye reactive kafka
Describe the bug
I'm getting an error at startup .
I've already tried without success:
- mp.messaging.incoming.data.tracing-enabled=false
Same for outgoing topic. The strangest thing about this is that I haven't been able to reproduce it locally (even in prod mode) and it only happens on a prod environment.
ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.lang.RuntimeException: Failed to start quarkus
at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
at io.quarkus.runtime.Application.start(Application.java:101)
at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
at io.quarkus.runner.GeneratedMain.main(Unknown Source)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:62)
at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [Le6zQbzkojAYO_OiKIQWJf4lGa4]: io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unexpected configuration error
at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.doCreate(Unknown Source)
at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
at io.quarkus.arc.runtime.BeanContainerImpl.beanInstance(BeanContainerImpl.java:26)
at io.quarkus.opentelemetry.runtime.tracing.intrumentation.InstrumentationRecorder.setupVertxTracer(InstrumentationRecorder.java:43)
at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy_0(Unknown Source)
at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy(Unknown Source)
... 13 more
Caused by: io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unexpected configuration error
at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:427)
at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:65)
at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:51)
at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.createSynthetic(Unknown Source)
... 32 more
Caused by: java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception.
at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:107)
at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.maybeSetAsGlobal(AutoConfiguredOpenTelemetrySdkBuilder.java:476)
at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:408)
... 35 more
Caused by: java.lang.Throwable
at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:115)
at io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:85)
at io.smallrye.reactive.messaging.kafka.tracing.KafkaOpenTelemetryInstrumenter.create(KafkaOpenTelemetryInstrumenter.java:41)
at io.smallrye.reactive.messaging.kafka.tracing.KafkaOpenTelemetryInstrumenter.createForSource(KafkaOpenTelemetryInstrumenter.java:27)
at io.smallrye.reactive.messaging.kafka.impl.KafkaSource.<init>(KafkaSource.java:230)
at io.smallrye.reactive.messaging.kafka.KafkaConnector.getPublisher(KafkaConnector.java:234)
at io.smallrye.reactive.messaging.kafka.KafkaConnector_Subclass.getPublisher$$superforward(Unknown Source)
at io.smallrye.reactive.messaging.kafka.KafkaConnector_Subclass$$function$$9.apply(Unknown Source)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
at io.quarkus.smallrye.reactivemessaging.runtime.DuplicatedContextConnectorFactoryInterceptor.intercept(DuplicatedContextConnectorFactoryInterceptor.java:32)
at io.quarkus.smallrye.reactivemessaging.runtime.DuplicatedContextConnectorFactoryInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
at io.smallrye.reactive.messaging.kafka.KafkaConnector_Subclass.getPublisher(Unknown Source)
at io.smallrye.reactive.messaging.kafka.KafkaConnector_ClientProxy.getPublisher(Unknown Source)
at io.smallrye.reactive.messaging.providers.impl.ConfiguredChannelFactory.createPublisher(ConfiguredChannelFactory.java:188)
at io.smallrye.reactive.messaging.providers.impl.ConfiguredChannelFactory.register(ConfiguredChannelFactory.java:142)
at io.smallrye.reactive.messaging.providers.impl.ConfiguredChannelFactory.initialize(ConfiguredChannelFactory.java:112)
at io.smallrye.reactive.messaging.providers.impl.ConfiguredChannelFactory_ClientProxy.initialize(Unknown Source)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
at io.smallrye.reactive.messaging.providers.extension.MediatorManager.start(MediatorManager.java:250)
at io.smallrye.reactive.messaging.providers.extension.MediatorManager_ClientProxy.start(Unknown Source)
at io.quarkus.smallrye.reactivemessaging.runtime.SmallRyeReactiveMessagingLifecycle.onApplicationStart(SmallRyeReactiveMessagingLifecycle.java:53)
at io.quarkus.smallrye.reactivemessaging.runtime.SmallRyeReactiveMessagingLifecycle_Observer_onApplicationStart_qTrMuLFyQ1IvGfeSxRVitl6CCBQ.notify(Unknown Source)
at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346)
at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:328)
at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:82)
at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:155)
at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:106)
at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
... 13 more
Expected behavior
No error at startup
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
17
Quarkus version or git rev
3.8.2
Build tool (ie. output of mvnw --version or gradlew --version)
No response
Additional information
No response
/cc @Ladicek (smallrye), @alesj (kafka), @brunobat (opentelemetry), @cescoffier (kafka), @jmartisk (smallrye), @ozangunalp (kafka), @phillip-kruger (smallrye), @radcortez (opentelemetry,smallrye)
![quarkus-bot[bot] avatar](https://avatars.githubusercontent.com/in/90234?v=4 "Published by a pub.dev verified publisher"){kind=small}
This bug should be filed in the smallrye reactive messaging repo: https://github.com/smallrye/smallrye-reactive-messaging For some weird reason, in your prod env, the static init is kicking earlier here: https://github.com/smallrye/smallrye-reactive-messaging/blob/570f8e802500b662b47f48e6cae53f0c5dc13422/smallrye-reactive-messaging-kafka/src/main/java/io/smallrye/reactive/messaging/kafka/tracing/KafkaOpenTelemetryInstrumenter.java#L41
We should not static init the instrumenters, the OpenTelemetry object should be injected because it's created here: https://github.com/quarkusio/quarkus/blob/079d8d08e929fd0de94fce42650b437b06a7ace6/extensions/opentelemetry/deployment/src/main/java/io/quarkus/opentelemetry/deployment/OpenTelemetryProcessor.java#L115
CC @ozangunalp
There is also the issue with tracing-enabled=false config not working for you. When not enabled that instrumenter isn't created at all.
This bug should be filed in the smallrye reactive messaging repo: https://github.com/smallrye/smallrye-reactive-messaging For some weird reason, in your prod env, the static init is kicking earlier here: https://github.com/smallrye/smallrye-reactive-messaging/blob/570f8e802500b662b47f48e6cae53f0c5dc13422/smallrye-reactive-messaging-kafka/src/main/java/io/smallrye/reactive/messaging/kafka/tracing/KafkaOpenTelemetryInstrumenter.java#L41
We should not static init the instrumenters, the OpenTelemetry object should be injected because it's created here:
https://github.com/quarkusio/quarkus/blob/079d8d08e929fd0de94fce42650b437b06a7ace6/extensions/opentelemetry/deployment/src/main/java/io/quarkus/opentelemetry/deployment/OpenTelemetryProcessor.java#L115
CC @ozangunalp
Should I open the same issue there or is it something that the quarkus team will address?
@itatdcer I am creating the issue upstream but I'd like to keep this one for tracking purposes.
I remember discussing this back then with @kabir, I think we still need a failover for creating the global OpenTelemetry instance because Reactive Messaging may run in an environment where it is not available as a CDI bean.
@itatdcer I am creating the issue upstream but I'd like to keep this one for tracking purposes.
I remember discussing this back then with @kabir, I think we still need a failover for creating the global OpenTelemetry instance because Reactive Messaging may run in an environment where it is not available as a CDI bean.
Makes sense.
Not sure if I understood correctly, @itatdcer... You say that when you set:
mp.messaging.incoming.data.tracing-enabled=false
The instrumenter is still instantiated and throws that error?... I would expect that the instrumenter would not be instantiated in that case.
Not sure if I understood correctly, @itatdcer... You say that when you set:
mp.messaging.incoming.data.tracing-enabled=falseThe instrumenter is still instantiated and throws that error?... I would expect that the instrumenter would not be instantiated in that case.
That's correct. Assuming that the configuration is a runtime one (I can't confirm it on quarkus documentation), I've tried that without success.
For now, I've changed the build time configuration:
quarkus.otel.enabled=false
and bypassed the error.
It would be nice to have this fix back ported to Quarkus 3.8 LTS. I know it should be done upstream, but I was wondering.
