grpc-java icon indicating copy to clipboard operation
grpc-java copied to clipboard
verified publisher icon grpc

BlockingStub call hangs indefinitely in ClientCalls$ThreadlessExecutor.waitAndDrain after server completion

Open halibobo1205 opened this issue 1 month ago • 11 comments

What version of gRPC-Java are you using?

gRPC-Java 1.75

What is your environment?

Linux arm64,JDK 17.0.15

What did you expect to see?

The blocking stub should return once the server sends a grpc-status: 0 completion, or throw a timeout exception if the deadline expires — it should never hang indefinitely.

What did you see instead?

When using a blocking stub with a timeout interceptor, the client occasionally hangs forever in:

io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:817)

even though the server has already completed the RPC successfully (grpc-status: 0). The thread remains parked indefinitely, and the application never proceeds.

This issue seems to occur randomly (low frequency)

Steps to reproduce the bug

###nterceptor

public class TimeoutInterceptor implements ClientInterceptor {

  private final long timeout;

  /**
   * @param timeout ms
   */
  public TimeoutInterceptor(long timeout) {
    this.timeout = timeout;
  }

  @Override
  public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
      MethodDescriptor<ReqT, RespT> method,
      CallOptions callOptions,
      Channel next) {
    return next.newCall(method,
        callOptions.withDeadlineAfter(timeout, TimeUnit.MILLISECONDS));
  }
}

stack

"Test worker" #1 prio=5 os_prio=0 cpu=68854.65ms elapsed=51528.48s tid=0x0000ffffb40311a0 nid=0x321d1c waiting on condition  [0x0000ffffb99fb000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
    - parking to wait for  <0x00000000d69d5a18> (a io.grpc.stub.ClientCalls$ThreadlessExecutor)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
    at io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:817)
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:166)
    at org.tron.api.DatabaseGrpc$DatabaseBlockingStub.getBlockByNum(DatabaseGrpc.java:343)
    at org.tron.common.client.DatabaseGrpcClient.getBlock(DatabaseGrpcClient.java:44)
    ...

grpc log

12:45:36.755 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[:authority: 127.0.0.1:57932, :path: /protocol.Database/GetBlockByNum, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.75.0, grpc-accept-encoding: gzip] padding=0 endStream=false
12:45:36.756 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] OUTBOUND DATA: streamId=5 padding=0 endStream=true length=5 bytes=0000000000
12:45:36.756 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] INBOUND HEADERS: streamId=5 headers=GrpcHttp2RequestHeaders[:path: /protocol.Database/GetBlockByNum, :authority: 127.0.0.1:57932, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.75.0, grpc-accept-encoding: gzip] padding=0 endStream=false
12:45:36.757 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] INBOUND DATA: streamId=5 padding=0 endStream=true length=5 bytes=0000000000
12:45:36.758 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip] padding=0 endStream=false
12:45:36.759 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND DATA: streamId=5 padding=0 endStream=false length=663 bytes=00000002920a720a705a6e0801126a0a2d747970652e676f6f676c65617069732e636f6d2f70726f746f636f6c2e5472616e73666572436f6e74726163741239...
12:45:36.759 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] padding=0 endStream=true
12:45:36.760 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip] padding=0 endStream=false
12:45:36.760 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND DATA: streamId=5 padding=0 endStream=false length=663 bytes=00000002920a720a705a6e0801126a0a2d747970652e676f6f676c65617069732e636f6d2f70726f746f636f6c2e5472616e73666572436f6e74726163741239...
12:45:36.761 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] padding=0 endStream=true


12:46:36.790 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND GO_AWAY: lastStreamId=2147483647 errorCode=0 length=8 bytes=6d61785f69646c65
12:46:36.791 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND PING: ack=false bytes=40715087873
12:46:36.791 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND GO_AWAY: lastStreamId=2147483647 errorCode=0 length=8 bytes=6d61785f69646c65
12:46:36.795 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND PING: ack=false bytes=40715087873
12:46:36.795 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] OUTBOUND PING: ack=true bytes=40715087873
12:46:36.795 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] INBOUND PING: ack=true bytes=40715087873
12:46:36.795 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND GO_AWAY: lastStreamId=5 errorCode=0 length=8 bytes=6d61785f69646c65
12:46:36.796 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND GO_AWAY: lastStreamId=5 errorCode=0 length=8 bytes=6d61785f69646c65

halibobo1205 avatar Oct 30 '25 10:10 halibobo1205

Do you see any log about uncaught exception from ThreadlessExecutor.runQuietly?

kannanjgithub avatar Nov 05 '25 11:11 kannanjgithub

@kannanjgithub No, I added a DebugInterceptor. When it gets stuck, there are no onMessage and onClose logs.

public class DebugInterceptor implements ClientInterceptor {
  @Override
  public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
      MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
    ClientCall<ReqT, RespT> call = next.newCall(method, callOptions);
    return new ForwardingClientCall.SimpleForwardingClientCall<ReqT, RespT>(call) {
      @Override
      public void start(Listener<RespT> responseListener, Metadata headers) {
        super.start(new ForwardingClientCallListener.SimpleForwardingClientCallListener<RespT>(
            responseListener) {
          @Override
          public void onHeaders(Metadata headers) {
            logger.debug("onHeaders: {}", headers);
            super.onHeaders(headers);
          }

          @Override
          public void onClose(Status status, Metadata trailers) {
            logger.debug("onClose: {}, {}", status, trailers);
            super.onClose(status, trailers);
          }

          @Override
          public void onMessage(RespT message) {
            logger.debug("onMessage: {}", message);
            super.onMessage(message);
          }
        }, headers);
      }
    };
  }
}

halibobo1205 avatar Nov 06 '25 11:11 halibobo1205

There was a fix we made for handling RST_STREAM but that already went into 1.74.0 and you are on 1.75.0. Can you enable io.netty debug logs as well on the client?

kannanjgithub avatar Nov 07 '25 06:11 kannanjgithub

@kannanjgithub, client and server both are debug logs

2025-11-07 17:12:54.849 DEBUG [Test worker] [i.n.u.i.InternalThreadLocalMap](InternalThreadLocalMap.java:100) -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2025-11-07 17:12:54.849 DEBUG [Test worker] [i.n.u.i.InternalThreadLocalMap](InternalThreadLocalMap.java:101) -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2025-11-07 17:12:54.861 DEBUG [Test worker] [i.n.c.n.NioEventLoop](NioEventLoop.java:110) -Dio.netty.noKeySetOptimization: false
2025-11-07 17:12:54.861 DEBUG [Test worker] [i.n.c.n.NioEventLoop](NioEventLoop.java:111) -Dio.netty.selectorAutoRebuildThreshold: 512
2025-11-07 17:12:54.904 DEBUG [Test worker] [i.n.u.i.PlatformDependent](PlatformDependent.java:1048) org.jctools-core.MpscChunkedArrayQueue: available
2025-11-07 17:13:22.506 DEBUG [Test worker] [i.n.u.ResourceLeakDetector](ResourceLeakDetector.java:129) -Dio.netty.leakDetection.level: simple
2025-11-07 17:13:22.507 DEBUG [Test worker] [i.n.u.ResourceLeakDetector](ResourceLeakDetector.java:130) -Dio.netty.leakDetection.targetRecords: 4
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:161) -Dio.netty.allocator.numHeapArenas: 16
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:162) -Dio.netty.allocator.numDirectArenas: 16
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:164) -Dio.netty.allocator.pageSize: 8192
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:169) -Dio.netty.allocator.maxOrder: 9
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:173) -Dio.netty.allocator.chunkSize: 4194304
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:174) -Dio.netty.allocator.smallCacheSize: 256
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:175) -Dio.netty.allocator.normalCacheSize: 64
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:176) -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:177) -Dio.netty.allocator.cacheTrimInterval: 8192
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:178) -Dio.netty.allocator.cacheTrimIntervalMillis: 0
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:179) -Dio.netty.allocator.useCacheForAllThreads: false
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:180) -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
2025-11-07 17:13:22.509 DEBUG [Test worker] [i.n.b.PooledByteBufAllocator](PooledByteBufAllocator.java:182) -Dio.netty.allocator.disableCacheFinalizersForFastThreadLocalThreads: false

halibobo1205 avatar Nov 07 '25 09:11 halibobo1205

There is not much in these logs. We need logs like in this comment here with details about the netty streams.

kannanjgithub avatar Nov 07 '25 10:11 kannanjgithub

@kannanjgithub How do I configure log output if there are no similar logs during the call? Would you like me to configure something like a LoggingHandler?

halibobo1205 avatar Nov 10 '25 03:11 halibobo1205

By the way, this issue doesn't happen on x86.

halibobo1205 avatar Nov 10 '25 03:11 halibobo1205

12:45:36.760 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND DATA: streamId=5 padding=0 endStream=false length=663 bytes=00000002920a720a705a6e0801126a0a2d747970652e676f6f676c65617069732e636f6d2f70726f746f636f6c2e5472616e73666572436f6e74726163741239... 12:45:36.761 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] padding=0 endStream=true

@kannanjgithub Hi, from the grpc logs mentioned above, does the logs as "INBOUND DATA" mean that the response data from the server has been transferred to the grpc-core, such as the grpc netty client?

waynercheung avatar Nov 10 '25 07:11 waynercheung 

2025-11-11 07:59:23.821 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0x457f0d6d, L:/127.0.0.1:15000 - R:/127.0.0.1:404
24] OUTBOUND HEADERS: streamId=33 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip] padd
ing=0 endStream=false
2025-11-11 07:59:23.821 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0x457f0d6d, L:/127.0.0.1:15000 - R:/127.0.0.1:404
24] OUTBOUND DATA: streamId=33 padding=0 endStream=false length=99 bytes=000000005e225c10021a58436f6e74726163742076616c6964617465206572726f72203a20636f6e74726163742074797
065206572726f722c20756e65787065...
2025-11-11 07:59:23.821 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0x457f0d6d, L:/127.0.0.1:15000 - R:/127.0.0.1:404
24] OUTBOUND HEADERS: streamId=33 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] padding=0 endStream=true
2025-11-11 07:59:23.822 DEBUG [grpc-nio-worker-ELG-2-4] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0x04ab8366, L:/127.0.0.1:40424 - R:/127.0.0.1:150
00] INBOUND HEADERS: streamId=33 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip] paddi
ng=0 endStream=false
2025-11-11 07:59:23.822 DEBUG [grpc-nio-worker-ELG-2-4] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0x04ab8366, L:/127.0.0.1:40424 - R:/127.0.0.1:150
00] INBOUND DATA: streamId=33 padding=0 endStream=false length=99 bytes=000000005e225c10021a58436f6e74726163742076616c6964617465206572726f72203a20636f6e747261637420747970
65206572726f722c20756e65787065...
2025-11-11 07:59:23.823 DEBUG [grpc-nio-worker-ELG-2-4] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0x04ab8366, L:/127.0.0.1:40424 - R:/127.0.0.1:150
00] INBOUND HEADERS: streamId=33 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] padding=0 endStream=true
2025-11-11 08:00:23.764 DEBUG [grpc-nio-worker-ELG-2-6] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0xcd83b03b, L:/127.0.0.1:30265 - R:/127.0.0.1:576
16] OUTBOUND GO_AWAY: lastStreamId=2147483647 errorCode=0 length=8 bytes=6d61785f69646c65
2025-11-11 08:00:23.766 DEBUG [grpc-nio-worker-ELG-2-6] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0xcd83b03b, L:/127.0.0.1:30265 - R:/127.0.0.1:576
16] OUTBOUND PING: ack=false bytes=40715087873
2025-11-11 08:00:23.766 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0xd233fc65, L:/127.0.0.1:57616 - R:/127.0.0.1:302
65] INBOUND GO_AWAY: lastStreamId=2147483647 errorCode=0 length=8 bytes=6d61785f69646c65
2025-11-11 08:00:23.768 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.ChannelLogger](ChannelTracer.java:131) [Subchannel<22>: (127.0.0.1:30265)] NettyClientTransport<23>: (/127.0.
0.1:30265) SHUTDOWN with UNAVAILABLE(GOAWAY shut down transport. HTTP/2 error code: NO_ERROR, debug data: max_idle)
2025-11-11 08:00:23.769 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.ChannelLogger](ChannelTracer.java:131) [Channel<3>: (127.0.0.1:30265)] Entering IDLE state with picker: io.gr
pc.internal.PickFirstLoadBalancer$RequestConnectionPicker@2c40c52b
2025-11-11 08:00:23.769 DEBUG [grpc-default-executor-1] [i.g.i.DnsNameResolver](DnsNameResolver.java:305) Attempting DNS resolution of 127.0.0.1
2025-11-11 08:00:23.770 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0xd233fc65, L:/127.0.0.1:57616 - R:/127.0.0.1:302
65] INBOUND PING: ack=false bytes=40715087873
2025-11-11 08:00:23.770 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0xd233fc65, L:/127.0.0.1:57616 - R:/127.0.0.1:302
65] OUTBOUND PING: ack=true bytes=40715087873
2025-11-11 08:00:23.770 DEBUG [grpc-nio-worker-ELG-2-6] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0xcd83b03b, L:/127.0.0.1:30265 - R:/127.0.0.1:576
16] INBOUND PING: ack=true bytes=40715087873


2025-11-11 08:00:23.771 DEBUG [grpc-nio-worker-ELG-2-6] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0xcd83b03b, L:/127.0.0.1:30265 - R:/127.0.0.1:576
16] OUTBOUND GO_AWAY: lastStreamId=3 errorCode=0 length=8 bytes=6d61785f69646c65

Hi, these are all the logs we have. @kannanjgithub

halibobo1205 avatar Nov 13 '25 11:11 halibobo1205

@kannanjgithub Hi, from the grpc logs mentioned above, does the logs as "INBOUND DATA" mean that the response data from the server has been transferred to the grpc-core, such as the grpc netty client?

Yes.

We are still looking into the issue.

kannanjgithub avatar Nov 14 '25 11:11 kannanjgithub

Hang observed on:

  • Amazon Linux 2023.6.20241121
  • Amazon Linux 2023.7.20250527
  • Amazon Linux 2023.7.20250623
  • Amazon Linux 2023.4.20240611

There are still no other useful logs.

2025-11-20 14:49:44.098 DEBUG [grpc-nio-worker-ELG-2-1] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0x6ca6a5c9, L:/127.0.0.1:15305 - R:/127.0.0.1:57270] OUTBOUND HEADERS: streamId=39 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip] padding=0 endStream=false
2025-11-20 14:49:44.099 DEBUG [grpc-nio-worker-ELG-2-1] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0x6ca6a5c9, L:/127.0.0.1:15305 - R:/127.0.0.1:57270] OUTBOUND DATA: streamId=39 padding=0 endStream=false length=5 bytes=0000000000
2025-11-20 14:49:44.099 DEBUG [grpc-nio-worker-ELG-2-1] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0x6ca6a5c9, L:/127.0.0.1:15305 - R:/127.0.0.1:57270] OUTBOUND HEADERS: streamId=39 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] padding=0 endStream=true
2025-11-20 14:49:44.099 DEBUG [grpc-nio-worker-ELG-2-16] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0x81e635dc, L:/127.0.0.1:57270 - R:/127.0.0.1:15305] INBOUND HEADERS: streamId=39 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip] padding=0 endStream=false
2025-11-20 14:49:44.099 DEBUG [grpc-nio-worker-ELG-2-16] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0x81e635dc, L:/127.0.0.1:57270 - R:/127.0.0.1:15305] INBOUND DATA: streamId=39 padding=0 endStream=false length=5 bytes=0000000000
2025-11-20 14:49:44.099 DEBUG [grpc-nio-worker-ELG-2-16] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0x81e635dc, L:/127.0.0.1:57270 - R:/127.0.0.1:15305] INBOUND HEADERS: streamId=39 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] padding=0 endStream=true
2025-11-20 14:50:44.054 DEBUG [grpc-nio-worker-ELG-2-6] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0x0a325c07, L:/127.0.0.1:11901 - R:/127.0.0.1:59558] OUTBOUND GO_AWAY: lastStreamId=2147483647 errorCode=0 length=8 bytes=6d61785f69646c65
2025-11-20 14:50:44.056 DEBUG [grpc-nio-worker-ELG-2-6] [i.g.n.NettyServerHandler](AbstractInternalLogger.java:214) [id: 0x0a325c07, L:/127.0.0.1:11901 - R:/127.0.0.1:59558] OUTBOUND PING: ack=false bytes=40715087873
2025-11-20 14:50:44.056 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.n.NettyClientHandler](AbstractInternalLogger.java:214) [id: 0x80d08a8a, L:/127.0.0.1:59558 - R:/127.0.0.1:11901] INBOUND GO_AWAY: lastStreamId=2147483647 errorCode=0 length=8 bytes=6d61785f69646c65
2025-11-20 14:50:44.057 DEBUG [grpc-nio-worker-ELG-2-5] [i.g.ChannelLogger](ChannelTracer.java:131) [Subchannel<22>: (127.0.0.1:11901)] NettyClientTransport<23>: (/127.0.0.1:11901) SHUTDOWN with UNAVAILABLE(GOAWAY shut down transport. HTTP/2 error code: NO_ERROR, debug data: max_idle)

halibobo1205 avatar Nov 21 '25 10:11 halibobo1205