[CDCSDK] Data loss issue without tablet splitting with gRPC connector (2024.2.0.0-b75)

[shamanthchandra-yb]

Jira Link: DB-13551

Description

Two test runs have encountered data loss issues—one on the latest master and one on version 2024.2.0.0-b75.

Run links attached in JIRA.

test_cdc_comply_advantage_without_tablet_splitting

AssertionError: Change is the same for 30 minutes test_cdc_7831a7 yb=153764 sink=153755. Replication stopped at 2024-10-17T16:56:02.867970

test_cdc_lru_nemesis_postgres_debezium

AssertionError: Change is the same for 30 minutes test_cdc_1a7c43 yb=326303 sink=326294. Replication stopped at 2024-10-18T05:10:26.700719

Both tests should have been run without tablet splitting. For test_cdc_comply_advantage_without_tablet_splitting, we explicitly set tablet splitting to false. While we didn’t set it for test_cdc_lru_nemesis_postgres_debezium, tablet splitting is enabled by default now. However, I don’t believe tablet splitting is the cause of the issue, as no aggressive tablet splitting flags were set.

Source connector version

quay.io/yugabyte/debezium-connector:dz.1.9.5.yb.grpc.2024.1

Connector configuration

add connector connector_name='ybconnector_cdc_842846_test_cdc_6c9464' stream_id='621351fcad395abdf14530743264650f' db_name='cdc_842846' connector_host='172.151.23.60' table_list=['test_cdc_6c9464'] {'name': 'ybconnector_cdc_842846_test_cdc_6c9464', 'config': {'database.master.addresses': '172.151.29.163:7100,172.151.16.49:7100,172.151.25.12:7100', 'database.hostname': '172.151.29.163:5433,172.151.16.49:5433,172.151.25.12:5433', 'database.port': 5433, 'database.masterhost': '172.151.29.163', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_842846', 'snapshot.mode': 'initial', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 300000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 2, 'topic.creation.default.replication.factor': '1', 'tasks.max': '5', 'connector.class': 'io.debezium.connector.yugabytedb.YugabyteDBgRPCConnector', 'database.server.name': 'db_cdc', 'database.streamid': '621351fcad395abdf14530743264650f', 'table.include.list': 'public.test_cdc_6c9464', 'database.sslrootcert': '/kafka/ca.crt'}}

YugabyteDB version

2.25.0.0-b155

and

2024.2.0.0-b75

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.