keycloak
Failing operator test - KeycloakDeploymentTest.testPodNamePropagation
Description
In several job runs, inluding https://github.com/keycloak/keycloak/actions/runs/10594540860/job/29358667097, KeycloakDeploymentTest.testPodNamePropagation is failing due to an underlying exception of
Caused by: jakarta.enterprise.inject.UnsatisfiedResolutionException: No bean found for required type [class io.quarkus.opentelemetry.runtime.tracing.DelayedAttributes] and qualifiers [[@jakarta.enterprise.inject.Any()]]
This is not consistently occurring nor has it been reproducable locally. Locally I consistenly observe TracerProducer providing the DelayedAttributes regardless of where other breakpoints are set in attempts to alter the startup.
cc @geoand @mabartos
Quarkus 3.13.3 is used. The stack trace:
2024-08-28T10:29:28.5140978Z jakarta.enterprise.inject.CreationException: Error creating synthetic bean [Le6zQbzkojAYO_OiKIQWJf4lGa4]: jakarta.enterprise.inject.UnsatisfiedResolutionException: No bean found for required type [class io.quarkus.opentelemetry.runtime.tracing.DelayedAttributes] and qualifiers [[@jakarta.enterprise.inject.Any()]]
2024-08-28T10:29:28.5144299Z at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.doCreate(Unknown Source)
2024-08-28T10:29:28.5145539Z at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
2024-08-28T10:29:28.5146472Z at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
2024-08-28T10:29:28.5147378Z at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
2024-08-28T10:29:28.5148220Z at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
2024-08-28T10:29:28.5148940Z at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
2024-08-28T10:29:28.5149549Z at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
2024-08-28T10:29:28.5150412Z at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
2024-08-28T10:29:28.5151298Z at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
2024-08-28T10:29:28.5152191Z at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
2024-08-28T10:29:28.5153211Z at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
2024-08-28T10:29:28.5154201Z at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
2024-08-28T10:29:28.5155119Z at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
2024-08-28T10:29:28.5155843Z at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
2024-08-28T10:29:28.5156513Z at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
2024-08-28T10:29:28.5157275Z at io.quarkus.arc.runtime.BeanContainerImpl.beanInstance(BeanContainerImpl.java:26)
2024-08-28T10:29:28.5158477Z at io.quarkus.opentelemetry.runtime.tracing.intrumentation.InstrumentationRecorder.setupVertxTracer(InstrumentationRecorder.java:47)
2024-08-28T10:29:28.5159990Z at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy_0(Unknown Source)
2024-08-28T10:29:28.5160875Z at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy(Unknown Source)
2024-08-28T10:29:28.5161699Z at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
2024-08-28T10:29:28.5162253Z at io.quarkus.runtime.Application.start(Application.java:101)
2024-08-28T10:29:28.5162955Z at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:119)
2024-08-28T10:29:28.5163599Z at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
2024-08-28T10:29:28.5164154Z at org.keycloak.quarkus.runtime.KeycloakMain.start(KeycloakMain.java:141)
2024-08-28T10:29:28.5164959Z at org.keycloak.quarkus.runtime.cli.command.AbstractStartCommand.run(AbstractStartCommand.java:42)
2024-08-28T10:29:28.5165713Z at picocli.CommandLine.executeUserObject(CommandLine.java:2030)
2024-08-28T10:29:28.5166513Z at picocli.CommandLine.access$1500(CommandLine.java:148)
2024-08-28T10:29:28.5167230Z at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
2024-08-28T10:29:28.5167959Z at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
2024-08-28T10:29:28.5168472Z at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
2024-08-28T10:29:28.5169077Z at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
2024-08-28T10:29:28.5169696Z at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
2024-08-28T10:29:28.5170186Z at picocli.CommandLine.execute(CommandLine.java:2174)
2024-08-28T10:29:28.5170750Z at org.keycloak.quarkus.runtime.cli.Picocli.parseAndRun(Picocli.java:138)
2024-08-28T10:29:28.5171422Z at org.keycloak.quarkus.runtime.KeycloakMain.main(KeycloakMain.java:101)
2024-08-28T10:29:28.5172237Z at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2024-08-28T10:29:28.5173001Z at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-08-28T10:29:28.5173647Z at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:62)
2024-08-28T10:29:28.5174557Z at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)
2024-08-28T10:29:28.5175950Z Caused by: jakarta.enterprise.inject.UnsatisfiedResolutionException: No bean found for required type [class io.quarkus.opentelemetry.runtime.tracing.DelayedAttributes] and qualifiers [[@jakarta.enterprise.inject.Any()]]
2024-08-28T10:29:28.5177216Z at io.quarkus.arc.impl.InstanceImpl.bean(InstanceImpl.java:288)
2024-08-28T10:29:28.5177816Z at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
2024-08-28T10:29:28.5178605Z at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
2024-08-28T10:29:28.5179772Z at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer$TracingResourceCustomizer$1.apply(AutoConfiguredOpenTelemetrySdkBuilderCustomizer.java:71)
2024-08-28T10:29:28.5181502Z at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer$TracingResourceCustomizer$1.apply(AutoConfiguredOpenTelemetrySdkBuilderCustomizer.java:65)
2024-08-28T10:29:28.5183093Z at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.lambda$mergeCustomizer$13(AutoConfiguredOpenTelemetrySdkBuilder.java:648)
2024-08-28T10:29:28.5184349Z at io.opentelemetry.sdk.autoconfigure.ResourceConfiguration.configureResource(ResourceConfiguration.java:111)
2024-08-28T10:29:28.5185537Z at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:439)
2024-08-28T10:29:28.5186619Z at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:81)
2024-08-28T10:29:28.5187491Z at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:54)
2024-08-28T10:29:28.5188394Z at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.createSynthetic(Unknown Source)
2024-08-28T10:29:28.5190127Z ... 39 more
@brunobat https://github.com/keycloak/keycloak/blob/ecbd856176fa81203be0fe07464299d39883e9f8/operator/src/test/java/org/keycloak/operator/testsuite/integration/KeycloakDeploymentTest.java#L391
The only thing novel there vs the other operator tests are the Keycloak settings tracing-enabled=true (toggles quarkus.otel.traces.enabled to true) and log-level=io.opentelemetry:fine.
There are similar tests running against an in-vm keycloak https://github.com/keycloak/keycloak/blob/ecbd856176fa81203be0fe07464299d39883e9f8/quarkus/tests/integration/src/test/java/org/keycloak/it/cli/dist/TracingDistTest.java#L60 - but those haven't exhibited the same flakiness.
I am pretty sure we can there a timing issue here that can be resolved by changing the way DelayedAttributes are handled.
@brunobat the upgrade to 3.14.2 is still in progress: #32519
Once that is completed we'll be able to see if the issue reoccurs in github actions - it wasn't reproducible for me locally.
@brunobat the fix does not appear to have been successful. We have disabled the affected test: https://github.com/keycloak/keycloak/pull/33118/files#diff-66841197b0c15bb09bad83420bdbc5fcf04bc718225c5a3fb0dd10e4c7fa311fR393
cc @jonkoops @geoand
@mabartos from some of the discussion on Slack I heard you were working on this, so I will assign you. Let me know if this is not the case.
Not a release blocker as it affects only a preview feature (OTEL Tracing).
