amazon-kinesis-client
amazon-kinesis-client copied to clipboard
Constantly getting Http2ConnectionTerminatingException using KCL 2.2.7
Hi,
I'm using KCL and enhanced fan out to build a consumer. The consumer does some heavy lift and sometimes it could take over 30 seconds to process a batch of records. I found that when the record batch is large, I'm constantly getting Http2ConnectionTerminatingException. Just wonder if there is any configuration I can do to get rid of this exception.
full stack trace: io.netty.channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. software.amazon.awssdk.http.nio.netty.internal.http2.Http2ConnectionTerminatingException: An I/O error occurred on an associated Http2 stream [id: 0x00ddcc50, L:/192.168.15.97:61772 ! R:kinesis.us-west-2.amazonaws.com/34.223.45.98:443](H2 - 5) at software.amazon.awssdk.http.nio.netty.internal.http2.Http2StreamExceptionHandler.exceptionCaught(Http2StreamExceptionHandler.java:50) [AwsJavaSdk-HttpClient-NettyNioClient-2.0.jar:?]
(aws-java-sdk-NettyEventLoop-1-1) io.netty.channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. software.amazon.awssdk.http.nio.netty.internal.http2.Http2ConnectionTerminatingException: An I/O error occurred on an associated Http2 stream [id: 0x00ddcc50, L:/192.168.15.97:61772 ! R:kinesis.us-west-2.amazonaws.com/34.223.45.98:443](H2 - 5) at software.amazon.awssdk.http.nio.netty.internal.http2.Http2StreamExceptionHandler.exceptionCaught(Http2StreamExceptionHandler.java:50) [AwsJavaSdk-HttpClient-NettyNioClient-2.0.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273) [netty-all-4.1.jar:4.1.48.Final] at software.amazon.awssdk.http.nio.netty.internal.UnusedChannelExceptionHandler.exceptionCaught(UnusedChannelExceptionHandler.java:52) [AwsJavaSdk-HttpClient-NettyNioClient-2.0.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:98) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.ReadTimeoutHandler.channelIdle(ReadTimeoutHandler.java:90) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.jar:4.1.48.Final] at java.lang.Thread.run(Thread.java:834) [?:
Is this affecting your propagation delay of the messages or the millisbehindlatest of the shards?
I have the same issue, last 24 hours I've seen this message 12 times, before upgrading from 2.2.5 to 2.3.3 it never happened.
Any updates? I'm seeing the same exception
the same
+1. I'm also getting the same exception.
We're having the same issue, any updates?
We also see this issue. Restarting the consumers does not help. KCL: 2.3.10 SDK: 2.17.101
wow, I am using latest version 2.4.1 and also seeing this issue... looks like this is open since Nov/2020 and no fix or solution.
Got the same exception running on version 2.4.2, so will it casuse any data lose or some other potential data integrity issues