3.4.19 + Ktor OpenTelemetry - ClassCastException

[omni-htg]

Dear AIJava team,

After updating from 3.4.18 to 3.4.19, we started experiencing the following exception no matter what combination of dependency versions we use:

Exception in thread "main" java.lang.ClassCastException: class io.opentelemetry.sdk.metrics.SdkDoubleHistogram$SdkDoubleHistogramBuilder cannot be cast to class io.opentelemetry.javaagent.shaded.io.opentelemetry.extension.incubator.metrics.ExtendedDoubleHistogramBuilder (io.opentelemetry.sdk.metrics.SdkDoubleHistogram$SdkDoubleHistogramBuilder is in unnamed module of loader io.opentelemetry.javaagent.bootstrap.AgentClassLoader @45c8e616; io.opentelemetry.javaagent.shaded.io.opentelemetry.extension.incubator.metrics.ExtendedDoubleHistogramBuilder is in unnamed module of loader 'bootstrap')
2023-12-23 23:47:12     at io.opentelemetry.javaagent.instrumentation.opentelemetryapi.v1_31.metrics.ApplicationDoubleHistogramBuilder131.setAttributesAdvice(ApplicationDoubleHistogramBuilder131.java:39)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.instrumenter.http.HttpMetricsAdvice.applyOldServerDurationAdvice(HttpMetricsAdvice.java:102)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.instrumenter.http.HttpServerMetrics.<init>(HttpServerMetrics.java:70)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.instrumenter.InstrumenterBuilder.buildOperationListeners(InstrumenterBuilder.java:320)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.instrumenter.Instrumenter.<init>(Instrumenter.java:92)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.instrumenter.PropagatingFromUpstreamInstrumenter.<init>(PropagatingFromUpstreamInstrumenter.java:21)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.instrumenter.InstrumenterBuilder$InstrumenterConstructor.lambda$propagatingFromUpstream$1(InstrumenterBuilder.java:387)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.instrumenter.InstrumenterBuilder.buildInstrumenter(InstrumenterBuilder.java:284)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.instrumenter.InstrumenterBuilder.buildUpstreamInstrumenter(InstrumenterBuilder.java:243)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.instrumenter.InstrumenterBuilder$1.buildUpstreamInstrumenter(InstrumenterBuilder.java:399)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.api.internal.InstrumenterUtil.buildUpstreamInstrumenter(InstrumenterUtil.java:58)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.ktor.v2_0.server.KtorServerTracing$Feature.install(KtorServerTracing.kt:131)
2023-12-23 23:47:12     at io.opentelemetry.instrumentation.ktor.v2_0.server.KtorServerTracing$Feature.install(KtorServerTracing.kt:100)
2023-12-23 23:47:12     at io.ktor.server.application.ApplicationPluginKt.install(ApplicationPlugin.kt:100)
2023-12-23 23:47:12     at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
2023-12-23 23:47:12     at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2023-12-23 23:47:12     at kotlin.reflect.jvm.internal.calls.CallerImpl$Method.callMethod(CallerImpl.kt:97)
2023-12-23 23:47:12     at kotlin.reflect.jvm.internal.calls.CallerImpl$Method$Static.call(CallerImpl.kt:106)
2023-12-23 23:47:12     at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod$kotlin_reflection(KCallableImpl.kt:207)
2023-12-23 23:47:12     at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:112)
2023-12-23 23:47:12     at io.ktor.server.engine.internal.CallableUtilsKt.callFunctionWithInjection(CallableUtils.kt:119)
2023-12-23 23:47:12     at io.ktor.server.engine.internal.CallableUtilsKt.executeModuleFunction(CallableUtils.kt:36)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$launchModuleByName$1.invoke(ApplicationEngineEnvironmentReloading.kt:332)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$launchModuleByName$1.invoke(ApplicationEngineEnvironmentReloading.kt:331)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartupFor(ApplicationEngineEnvironmentReloading.kt:356)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.launchModuleByName(ApplicationEngineEnvironmentReloading.kt:331)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.access$launchModuleByName(ApplicationEngineEnvironmentReloading.kt:32)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:312)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:310)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartup(ApplicationEngineEnvironmentReloading.kt:338)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.instantiateAndConfigureApplication(ApplicationEngineEnvironmentReloading.kt:310)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createApplication(ApplicationEngineEnvironmentReloading.kt:150)
2023-12-23 23:47:12     at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.start(ApplicationEngineEnvironmentReloading.kt:277)
2023-12-23 23:47:12     at io.ktor.server.netty.NettyApplicationEngine.start(NettyApplicationEngine.kt:216)
2023-12-23 23:47:12     at io.ktor.server.netty.EngineMain.main(EngineMain.kt:23)

In our system we are using:

• AI Java as an agent, with the AI ConnectionString set up as an envvar: APPLICATIONINSIGHTS_CONNECTION_STRING.
• Kotlin 1.9.22
• Ktor 2.3.7
• OpenTelemetry Ktor 1.32.0-alpha, initialized in this way as a plugin:

    install(KtorServerTracing){
        setOpenTelemetry(GlobalOpenTelemetry.get())
    }

Rolling back to 3.4.18 (even without modifying any of the other dependencies) makes it work perfectly again.
Additionally we have found this issue #3151 which seems to be related, but because it's older we wanted to confirm before we make the jump in how OpenTelemetry is set up.

Thank you kindly!

[jeanbisutti]

@omni-htg Instead of adding and configuring the Ktor instrumentation library, you can enable the Ktor automatic instrumenation provided by Application Insights:

{
  "preview": {
    "instrumentation": {
      "ktor": {
        "enabled": true
      }
    }
  }
}

Please let us know if it works for you.

[omni-htg]

Thank you, @jeanbisutti !

I have tried using the APPLICATIONINSIGHTS_PREVIEW_INSTRUMENTATION_KTOR_ENABLED environment variable and removing the OpenTelemetry Ktor instrumentation library, which seems to work with 3.4.19.

I do have a couple of questions though:

• Is there a reason this stopped working from 18 to 19?
• How can I confirm that the automatic instrumentation is enabled?
• By removing the OpenTelemetry Ktor library, I lose access to the OpenTelemetry operations, like accessing TraceId or adding Custom Dimensions.

Thanks once again for your assistance.

[trask]

Is there a reason this stopped working from 18 to 19?

I suspect this may be a bug, fixed upstream already but not released yet: https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/10026

you can try to workaround the bug by adding this dependency to your application: io.opentelemetry:opentelemetry-sdk-extension-incubator:1.32.0-alpha

[omni-htg]

Hey @trask !

Just tried adding that package to the project and use 19 again -- same results. Thank you for trying!

If you have another workaround in mind, please let me know. Else, what is the ETA for that bugfix?

[omni-htg]

Hello team!

Is there any other input I could provide to help get to the bottom of this issue?

Thanks!