[CDCSDK] Regression: xCluster + CDC case failed with GCed issue

[shamanthchandra-yb]

Jira Link: DB-8051

Description

Please find run report in JIRA

2023-09-21 17:18:55,917 WARN   YugabyteDB|db_cdc|streaming|8  Throwing error with code: UNKNOWN_ERROR   [io.debezium.connector.yugabytedb.YugabyteDBStreamingChangeEventSource]
12364 2023-09-21 17:18:55,917 WARN   YugabyteDB|db_cdc|streaming|8  Error while trying to get the changes from the server; will attempt retry 1 of 10 after 60000 milli-seconds. Exception: {}   [io.debezium.connector.yugabytedb
12365 org.yb.client.CDCErrorException: Server[f78bd69db0644c74b155d1e289cec0e5] INTERNAL_ERROR[code 21]: CDCSDK Trying to fetch already GCed intents for transaction 9545dc55-581d-4a9f-b594-2152f9caac9a
12366         at org.yb.client.TabletClient.dispatchCDCErrorOrReturnException(TabletClient.java:506)
12367         at org.yb.client.TabletClient.decode(TabletClient.java:437)
12368         at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
12369         at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
12370         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
12371         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
12372         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
12373         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
12374         at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
12375         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
12376         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
12377         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
12378         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
12379         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
12380         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
12381         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
12382         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
12383         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
12384         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
12385         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
12386         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
12387         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
12388         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
12389         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
12390         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
12391         at java.base/java.lang.Thread.run(Thread.java:829)

Source connector version

1.9.5.y.35-SNAPSHOT

Connector configuration

add connector connector_name='ybconnector_cdc_86855f_test_cdc_fd3371' stream_id='4b8cf1eaa4a14ad3be8b9c783ce9b869' db_name='cdc_86855f' connector_host='172.151.21.8' table_list=['test_cdc_fd3371'] {'name': 'ybconnector_cdc_86855f_test_cdc_fd3371', 'config': {'connector.class': 'io.debezium.connector.yugabytedb.YugabyteDBConnector', 'database.hostname': '172.151.29.157', 'database.master.addresses': '172.151.21.232:7100,172.151.28.26:7100,172.151.29.157:7100', 'database.port': 5433, 'database.masterhost': '172.151.29.157', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_86855f', 'database.server.name': 'db_cdc', 'database.streamid': '4b8cf1eaa4a14ad3be8b9c783ce9b869', 'snapshot.mode': 'never', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 600000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 2, 'topic.creation.default.replication.factor': '1', 'tasks.max': '10', 'table.include.list': 'public.test_cdc_fd3371'}}

YugabyteDB version

2.19.3.0-b125

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.