amazon-kinesis-client Constantly getting Http2ConnectionTerminatingException using KCL 2.2.7

Constantly getting Http2ConnectionTerminatingException using KCL 2.2.7

Open shaozhaoyang opened this issue 3 years ago • 9 comments

Hi,

I'm using KCL and enhanced fan out to build a consumer. The consumer does some heavy lift and sometimes it could take over 30 seconds to process a batch of records. I found that when the record batch is large, I'm constantly getting Http2ConnectionTerminatingException. Just wonder if there is any configuration I can do to get rid of this exception.

full stack trace: io.netty.channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. software.amazon.awssdk.http.nio.netty.internal.http2.Http2ConnectionTerminatingException: An I/O error occurred on an associated Http2 stream [id: 0x00ddcc50, L:/192.168.15.97:61772 ! R:kinesis.us-west-2.amazonaws.com/34.223.45.98:443](H2 - 5) at software.amazon.awssdk.http.nio.netty.internal.http2.Http2StreamExceptionHandler.exceptionCaught(Http2StreamExceptionHandler.java:50) [AwsJavaSdk-HttpClient-NettyNioClient-2.0.jar:?]

(aws-java-sdk-NettyEventLoop-1-1) io.netty.channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. software.amazon.awssdk.http.nio.netty.internal.http2.Http2ConnectionTerminatingException: An I/O error occurred on an associated Http2 stream [id: 0x00ddcc50, L:/192.168.15.97:61772 ! R:kinesis.us-west-2.amazonaws.com/34.223.45.98:443](H2 - 5) at software.amazon.awssdk.http.nio.netty.internal.http2.Http2StreamExceptionHandler.exceptionCaught(Http2StreamExceptionHandler.java:50) [AwsJavaSdk-HttpClient-NettyNioClient-2.0.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273) [netty-all-4.1.jar:4.1.48.Final] at software.amazon.awssdk.http.nio.netty.internal.UnusedChannelExceptionHandler.exceptionCaught(UnusedChannelExceptionHandler.java:52) [AwsJavaSdk-HttpClient-NettyNioClient-2.0.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:98) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.ReadTimeoutHandler.channelIdle(ReadTimeoutHandler.java:90) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.jar:4.1.48.Final] at java.lang.Thread.run(Thread.java:834) [?: