yugabyte-db icon indicating copy to clipboard operation
yugabyte-db copied to clipboard

[CDCSDK] CDC consistency broke in one of the run with multiple runs single table bank workload (with connector restart)

Open shamanthchandra-yb opened this issue 1 year ago • 0 comments

Jira Link: DB-7275

Description

http://stress.dev.yugabyte.com/stress_test/2d43c7da-b443-438d-b790-83bdd137eb80

There are 5 tables on which its going on but CDC Consistency broke in 3 of the tables.

Not sure, but in last 5 mins, stop/start nodes nemesis had happened.

Source connector version

1.9.5.y.31-SNAPSHOT

Connector configuration

add connector connector_name='yugabyte_connector' stream_id='5edc1ede0c44453eb61ef2e039fd102f' db_name='cdc_3fc018' connector_host='172.151.18.183' table_list=['test_cdc_ee1efa', 'test_cdc_bfe003', 'test_cdc_17df21', 'test_cdc_8ecfe8', 'test_cdc_65e4a3'] {'name': 'yugabyte_connector', 'config': {'connector.class': 'io.debezium.connector.yugabytedb.YugabyteDBConnector', 'database.hostname': '172.151.27.49', 'database.master.addresses': '172.151.29.247:7100,172.151.27.49:7100,172.151.29.80:7100', 'database.port': 5433, 'database.masterhost': '172.151.27.49', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_3fc018', 'database.server.name': 'db_cdc', 'database.streamid': '5edc1ede0c44453eb61ef2e039fd102f', 'snapshot.mode': 'never', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 600000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 1, 'topic.creation.default.replication.factor': '1', 'tasks.max': '1', 'table.include.list': 'public.test_cdc_ee1efa,public.test_cdc_bfe003,public.test_cdc_17df21,public.test_cdc_8ecfe8,public.test_cdc_65e4a3', 'transforms': 'Reroute', 'transforms.Reroute.topic.regex': '(.*)', 'transforms.Reroute.topic.replacement': 'db_cdc_all_events', 'transforms.Reroute.type': 'io.debezium.transforms.ByLogicalTableRouter', 'transforms.Reroute.key.field.regex': 'db_cdc(.*)', 'transforms.Reroute.key.field.replacement': '$1', 'transaction.ordering': 'true', 'provide.transaction.metadata': 'true', 'max.poll.interval.ms': '5000'}}

YugabyteDB version

2.18.2.0-b57

Warning: Please confirm that this issue does not contain any sensitive information

  • [X] I confirm this issue does not contain any sensitive information.

shamanthchandra-yb avatar Jul 18 '23 07:07 shamanthchandra-yb