quarkus icon indicating copy to clipboard operation
quarkus copied to clipboard

HttpInstrumenterVertxTracer error when DefaultHttp2Connection onStreamClosed

Open naah69 opened this issue 8 months ago • 10 comments

Describe the bug

2025-04-15 20:26:53,062 ERROR io.netty.handler.codec.http2.DefaultHttp2Connection - Caught Throwable from listener onStreamClosed.
java.lang.NullPointerException: Cannot invoke "io.opentelemetry.context.Context.get(io.opentelemetry.context.ContextKey)" because "context" is null
        at io.opentelemetry.instrumentation.api.internal.HttpRouteState.fromContextOrNull(HttpRouteState.java:25)
        at io.opentelemetry.instrumentation.api.semconv.http.HttpServerRoute.update(HttpServerRoute.java:99)
        at io.quarkus.opentelemetry.runtime.tracing.intrumentation.vertx.HttpInstrumenterVertxTracer.sendResponse(HttpInstrumenterVertxTracer.java:103)
        at io.quarkus.opentelemetry.runtime.tracing.intrumentation.vertx.HttpInstrumenterVertxTracer.sendResponse(HttpInstrumenterVertxTracer.java:50)
        at io.quarkus.opentelemetry.runtime.tracing.intrumentation.vertx.OpenTelemetryVertxTracer.sendResponse(OpenTelemetryVertxTracer.java:47)
        at io.quarkus.opentelemetry.runtime.tracing.intrumentation.vertx.OpenTelemetryVertxTracer.sendResponse(OpenTelemetryVertxTracer.java:16)
        at io.quarkus.opentelemetry.runtime.tracing.intrumentation.vertx.OpenTelemetryVertxTracingFactory$VertxDelegator.sendResponse(OpenTelemetryVertxTracingFactory.java:78)
        at io.vertx.core.http.impl.Http2ServerStream.onClose(Http2ServerStream.java:235)
        at io.vertx.core.http.impl.Http2ConnectionBase.onStreamClosed(Http2ConnectionBase.java:157)
        at io.vertx.core.http.impl.VertxHttp2ConnectionHandler$1.onStreamClosed(VertxHttp2ConnectionHandler.java:93)
        at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:355)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1034)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:990)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:515)
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:521)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.doCloseStream(Http2ConnectionHandler.java:919)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.access$900(Http2ConnectionHandler.java:64)
        at io.netty.handler.codec.http2.Http2ConnectionHandler$2.operationComplete(Http2ConnectionHandler.java:632)
        at io.netty.handler.codec.http2.Http2ConnectionHandler$2.operationComplete(Http2ConnectionHandler.java:629)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
        at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
        at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
        at io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete(DelegatingChannelPromiseNotifier.java:52)
        at io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete(DelegatingChannelPromiseNotifier.java:31)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
        at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
        at io.netty.handler.codec.http2.Http2CodecUtil$SimpleChannelPromiseAggregator.tryPromise(Http2CodecUtil.java:380)
        at io.netty.handler.codec.http2.Http2CodecUtil$SimpleChannelPromiseAggregator.trySuccess(Http2CodecUtil.java:347)
        at io.netty.handler.codec.http2.Http2CodecUtil$SimpleChannelPromiseAggregator.trySuccess(Http2CodecUtil.java:259)
        at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
        at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:748)
        at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:303)
        at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:383)
        at io.netty.channel.epoll.AbstractEpollStreamChannel.writeBytesMultiple(AbstractEpollStreamChannel.java:307)
        at io.netty.channel.epoll.AbstractEpollStreamChannel.doWriteMultiple(AbstractEpollStreamChannel.java:512)
        at io.netty.channel.epoll.AbstractEpollStreamChannel.doWrite(AbstractEpollStreamChannel.java:424)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:929)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:893)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1369)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:935)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:921)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:907)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:195)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:941)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:921)
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:907)
        at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:966)
        at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:253)
        at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.checkFlush(VertxHttp2ConnectionHandler.java:247)
        at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.writeData(VertxHttp2ConnectionHandler.java:242)
        at io.vertx.core.http.impl.VertxHttp2Stream.doWriteData(VertxHttp2Stream.java:247)
        at io.vertx.core.http.impl.VertxHttp2Stream.lambda$writeData$7(VertxHttp2Stream.java:212)
        at io.vertx.core.http.impl.VertxHttp2Stream.lambda$queueForWrite$8(VertxHttp2Stream.java:229)
        at io.opentelemetry.javaagent.bootstrap.executors.ContextPropagatingRunnable.run(ContextPropagatingRunnable.java:37)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Expected behavior

fix NPE

Actual behavior

NullPointerException

How to Reproduce?

  1. open http2
  2. use opentelemetry
  3. test it

Output of uname -a or ver

No response

Output of java -version

21

Quarkus version or git rev

3.15.1

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

naah69 avatar Apr 16 '25 07:04 naah69

/cc @brunobat (opentelemetry), @radcortez (opentelemetry)

quarkus-bot[bot] avatar Apr 16 '25 07:04 quarkus-bot[bot]

Am I correct in assuming (based on the stacktrace) that you are using the OpenTelemetry agent?

geoand avatar Apr 16 '25 11:04 geoand

Am I correct in assuming (based on the stacktrace) that you are using the OpenTelemetry agent?我是否正确地假设(基于堆栈跟踪)您正在使用 OpenTelemetry 代理?

yes,we always use OpenTelemetry agent,but when we use http2,it error

naah69 avatar Apr 16 '25 13:04 naah69

@naah69, So you are mixing the OpenTelemetry agent and the Quarkus OpenTelemetry extension?

You should only use one and the recommended way is the extension: https://quarkus.io/guides/opentelemetry

brunobat avatar Apr 16 '25 13:04 brunobat

@naah69, So you are mixing the OpenTelemetry agent and the Quarkus OpenTelemetry extension?

You should only use one and the recommended way is the extension: https://quarkus.io/guides/opentelemetry

@brunobat

The reason that we use both of them is we has custom trace filter and we need some opentelemery class of vertx in extension. In my option,quarkus.otel.sdk.disabled=true won't open extension. Is this way mixing agent and extension?

naah69 avatar Apr 16 '25 13:04 naah69

@brunobat @geoand

Does quarkus OpenTelemetry extension(default,without other jars) support these libraries and frameworks in page and custom filter?

We use many libraries and frameworks in OpenTelemetry Instrumentation.If it supportes,I think that we can remove agent.

naah69 avatar Apr 16 '25 13:04 naah69

In my test,when I add opentelemetry-okhttp-3.0 to pom.xml and send request by okhttp,there is no traceparent in request header at the server side.

Does OpenTelemetry extension support standalone library instrumentation?

naah69 avatar Apr 17 '25 13:04 naah69

Given you seem to do something specific, it would immensely help if you could prepare a small reproducer with proper instructions on how to reproduce the problem.

gsmet avatar Apr 17 '25 17:04 gsmet

@naah69 We support tracing on these extensions, by using quarkus-opentelemetry: https://quarkus.io/guides/opentelemetry-tracing#quarkus-extensions-using-opentelemetry

I'm not sure why you need so send requests with okhttp when you have the quarkus-rest-client: https://quarkus.io/guides/rest-client...

I second Guillaume's comment about the need for a reproducer.

brunobat avatar Apr 21 '25 09:04 brunobat

@naah69 any luck with the reproducer?

geoand avatar Apr 29 '25 10:04 geoand

Closing for the lack of a reproducer

geoand avatar May 06 '25 06:05 geoand