reactor-netty
reactor-netty copied to clipboard
Published
20 hours ago •
reactor
reactor
HTTP/2 Benchmarks on reactor netty 2 don't work
Expected Behavior
Be able to run some Gatling HTTP/2 bench using reactor netty 2 and netty 5.
Actual Behavior
HTTP/2 gatling benchmarks are failing with errors like this:
> Request timeout to 127.0.0.1/127.0.0.1:8080 after 60000 ms 15535 (62.97%)
> i.n.c.ConnectTimeoutException: connection timed out: /127.0.0. 8816 (35.74%)
1:8080
> i.n.h.s.SslHandshakeTimeoutException: handshake timed out after 319 ( 1.29%)
r 10000ms
Steps to Reproduce
- clone https://github.com/reactor/benchmarks
- Install JDK19
- build (./gradlew build)
- on one console, start the reactor netty 2 frontend using HTTP/2 protocol:
java -DPROTOCOL=H2 -jar frontend-rn-20x/build/libs/frontend-rn-20x-1.0.0.jar
- on another console, run Gatling:
rm -rf test-reports/; java -DPROTOCOL=H2 -DURATION=10 -jar gatling/build/libs/gatling-1.0.0-all.jar application “test” TextPlain
After few minutes, some errors will be displayed like "handshake timed out".
While the test is running, jstat seems to indicate that the frontend is having some heap memory leak:
MC MU CCSC CCSU OC OU YGC FGC FGCT CGC CGCT GCT
24704.0 24333.0 2944.0 2781.2 155648.0 106984.5 10 0 0.000 6 0.007 0.028
24896.0 24493.3 2944.0 2781.7 172032.0 107161.0 21 0 0.000 6 0.007 0.051
24960.0 24632.7 2944.0 2783.1 507904.0 197459.5 34 0 0.000 6 0.007 0.160
24960.0 24636.7 2944.0 2784.0 1638400.0 754936.5 40 0 0.000 6 0.007 0.363
24960.0 24638.4 2944.0 2784.0 2744320.0 1550304.0 45 0 0.000 6 0.007 0.631
24960.0 24638.4 2944.0 2784.0 2670592.0 1759994.5 50 0 0.000 6 0.007 0.774
24960.0 24638.4 2944.0 2784.0 3190784.0 2045706.0 54 0 0.000 6 0.007 0.946
24960.0 24638.7 2944.0 2784.0 3739648.0 1920992.0 58 0 0.000 8 0.008 1.247
25088.0 24686.6 2944.0 2784.0 3833856.0 2117195.5 62 0 0.000 8 0.008 1.473
25088.0 24742.3 2944.0 2784.5 4947968.0 3147744.0 67 0 0.000 9 0.010 2.010
25152.0 24803.0 2944.0 2784.5 6152192.0 4818912.0 71 0 0.000 9 0.010 2.416
25152.0 24805.2 2944.0 2784.5 7303168.0 6281184.0 81 0 0.000 10 0.010 2.999
25152.0 24805.2 2944.0 2784.5 7565312.0 6135776.0 90 0 0.000 11 0.013 3.440
25152.0 24805.2 2944.0 2784.5 7393280.0 5818336.0 99 0 0.000 12 0.013 4.085
25152.0 24805.2 2944.0 2784.5 7049216.0 5103584.0 110 0 0.000 13 0.015 4.677
25280.0 24872.6 2944.0 2784.5 7946240.0 6973408.0 116 0 0.000 13 0.015 5.307
25280.0 24879.1 2944.0 2784.5 8318976.0 8315002.5 135 0 0.000 13 0.015 6.234
25280.0 24778.9 2944.0 2755.4 5996544.0 3319548.6 139 1 0.450 13 0.015 6.900
25280.0 24838.7 2944.0 2755.4 7487488.0 5935374.1 143 1 0.450 13 0.015 7.540
25280.0 24838.7 2944.0 2755.4 8355840.0 8354581.7 196 2 0.450 13 0.015 8.299
25280.0 24838.7 2944.0 2755.4 7073792.0 5211586.1 198 2 1.121 13 0.015 9.261
25280.0 24838.7 2944.0 2755.4 7946240.0 7765442.1 208 2 1.121 13 0.015 9.967
25280.0 24849.1 2944.0 2755.4 7348224.0 5662974.5 251 3 1.970 13 0.015 11.251
25344.0 24916.1 2944.0 2755.4 8237056.0 8235258.5 268 3 1.970 13 0.015 11.987
25344.0 24916.1 2944.0 2755.4 7946240.0 6956097.3 290 4 2.927 13 0.015 13.330
25344.0 24920.2 2944.0 2755.4 8364032.0 8363073.3 350 5 2.927 13 0.015 13.902
25344.0 24966.4 2944.0 2755.4 8359936.0 8358496.2 412 6 3.999 13 0.015 15.565
25344.0 24967.5 2944.0 2755.4 8364032.0 8363298.6 456 7 5.088 13 0.015 17.063
25344.0 24967.5 2944.0 2755.4 8310784.0 8309361.3 458 8 6.204 13 0.015 18.271
25344.0 24969.1 2944.0 2755.4 8318976.0 8314907.8 462 10 8.350 13 0.015 20.592
25344.0 25008.7 2944.0 2760.2 8318976.0 8315733.3 464 11 9.458 13 0.015 21.802
Using the following Netty5 debug options don't report anything:
-Dio.netty5.leakDetection.level=paranoid
-Dio.netty5.leakDetection.targetRecords=32
-Dio.netty5.buffer.lifecycleTracingEnabled=true
-Dio.netty5.buffer.leakDetectionEnabled=true
Possible Solution
Your Environment
Java 19 Benchmarks project Reactor Netty 2 Netty 5
some updates: the frontend test server was not setting any max concurrent streams, and specify a max-concurrent-stream on the frontend server avoids the memory leak.
Now, the benchmark still fails with this exception that happens on the frontend server (using the "TextPlain" scenario:
16:35:04.683 [reactor-http-nio-10] WARN i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty5.handler.codec.http2.Http2Exception: Stream 13973 does not exist
at io.netty5.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:99)
at io.netty5.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.verifyStreamMayHaveExisted(DefaultHttp2ConnectionDecoder.java:667)
at io.netty5.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:427)
at io.netty5.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:513)
at io.netty5.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:256)
at io.netty5.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
at io.netty5.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174)
at io.netty5.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61)
at io.netty5.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
at io.netty5.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
at io.netty5.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:387)
at io.netty5.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:330)
at io.netty5.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:204)
at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
at io.netty5.channel.DefaultChannelHandlerContext.findAndInvokeChannelRead(DefaultChannelHandlerContext.java:445)
at io.netty5.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:426)
at reactor.netty5.http.server.HttpServerConfig$H2ChannelMetricsHandler.channelRead(HttpServerConfig.java:717)
at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
at io.netty5.channel.DefaultChannelHandlerContext.findAndInvokeChannelRead(DefaultChannelHandlerContext.java:445)
at io.netty5.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:426)
at io.netty5.channel.internal.DelegatingChannelHandlerContext.fireChannelRead(DelegatingChannelHandlerContext.java:113)
at io.netty5.handler.codec.ByteToMessageDecoder$ByteToMessageDecoderContext.fireChannelRead(ByteToMessageDecoder.java:446)
at io.netty5.handler.ssl.SslHandler.unwrap(SslHandler.java:1215)
at io.netty5.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1081)
at io.netty5.handler.ssl.SslHandler.decode(SslHandler.java:1128)
at io.netty5.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:387)
at io.netty5.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:330)
at io.netty5.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:204)
at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
at io.netty5.channel.DefaultChannelHandlerContext.findAndInvokeChannelRead(DefaultChannelHandlerContext.java:445)
at io.netty5.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:426)
at io.netty5.channel.ChannelHandler.channelRead(ChannelHandler.java:235)
at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
at io.netty5.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:838)
at io.netty5.channel.AbstractChannel$ReadSink.processRead(AbstractChannel.java:1996)
at io.netty5.channel.nio.AbstractNioByteChannel.doReadNow(AbstractNioByteChannel.java:74)
at io.netty5.channel.AbstractChannel$ReadSink.readLoop(AbstractChannel.java:2061)
at io.netty5.channel.AbstractChannel.readNow(AbstractChannel.java:910)
at io.netty5.channel.nio.AbstractNioChannel.access$100(AbstractNioChannel.java:42)
at io.netty5.channel.nio.AbstractNioChannel$1.handle(AbstractNioChannel.java:107)
at io.netty5.channel.nio.NioHandler.processSelectedKey(NioHandler.java:506)
at io.netty5.channel.nio.NioHandler.processSelectedKeysOptimized(NioHandler.java:489)
at io.netty5.channel.nio.NioHandler.processSelectedKeys(NioHandler.java:430)
at io.netty5.channel.nio.NioHandler.run(NioHandler.java:407)
at io.netty5.channel.SingleThreadEventLoop.runIO(SingleThreadEventLoop.java:192)
at io.netty5.channel.SingleThreadEventLoop.run(SingleThreadEventLoop.java:176)
at io.netty5.util.concurrent.SingleThreadEventExecutor.lambda$doStartThread$4(SingleThreadEventExecutor.java:773)
at io.netty5.util.internal.ThreadExecutorMap.lambda$apply$1(ThreadExecutorMap.java:68)
at io.netty5.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1589)