[CDCSDK] Saw a data loss issue with xcluster + CDC on master (2.21.0.0-b57)

[shamanthchandra-yb]

Jira Link: DB-8312

Description

Please refer JIRA for stress run report.

We were seeing GCed issue mainly as recent regression. But, data loss wasn't seen for this case.

Another doubt we had was whether "It could be just some timing issue with bootstrap as only 3 rows are missing in the first iteration itself."

But looks like No, because, There was 3 minutes gap between setting up and loading rows. So, suspecting if that’s not the case.

I see

2023-10-07 21:59:33,260 INFO   YugabyteDB|db_cdc|streaming|2  Requesting changes for tablet 5e243708873d4714b63e030d0becfcf8 from OpId term=1, index=2, key=, write_id=0, time=6949748624405291008 for table test_cdc_0ebb55   [io.debezium.connector.yugabytedb.YugabyteDBStreamingChangeEventSource]

logs, much before loading.

No tablet splitting has also happened, so eliminating a doubt with tablet splitting as well.

Source connector version

1.9.5.y.33-SNAPSHOT

Connector configuration

add connector connector_name='ybconnector_cdc_4a5a00_test_cdc_0ebb55' stream_id='40d6e546235940d187d703675526ea77' db_name='cdc_4a5a00' connector_host='172.151.23.71' table_list=['test_cdc_0ebb55'] {'name': 'ybconnector_cdc_4a5a00_test_cdc_0ebb55', 'config': {'connector.class': 'io.debezium.connector.yugabytedb.YugabyteDBConnector', 'database.hostname': '172.151.30.35:5433,172.151.17.225:5433,172.151.19.108:5433', 'database.master.addresses': '172.151.30.35:7100,172.151.17.225:7100,172.151.19.108:7100', 'database.masterhost': '172.151.30.35', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_4a5a00', 'database.server.name': 'db_cdc', 'database.streamid': '40d6e546235940d187d703675526ea77', 'snapshot.mode': 'never', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 600000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 2, 'topic.creation.default.replication.factor': '1', 'tasks.max': '10', 'table.include.list': 'public.test_cdc_0ebb55'}}

YugabyteDB version

2.21.0.0-b57

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.