[CDCSDK] Increased bootstrap time when there is connector lag

[shamanthchandra-yb]

Jira Link: DB-13431

Description

In CDC long retention setup, brought down the connector for ~9 to 10 hrs and did a rolling restart.

It took 2 hours, 10 minutes to bootstrap on a node. YBA task failed.

Please check JIRA for slack thread.

On a happy path it has taken between 10 and 20 mins currently

Source connector version

quay.io/yugabyte/debezium-connector:dz.1.9.5.yb.grpc.2024.1

Connector configuration

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" \
  localhost:8083/connectors/ \
  -d '{
    "name": "ybconnector_cdc_cc3b35_test_cdc_238adc",
    "config": {
      "database.master.addresses": "172.151.24.238:7100,172.151.23.56:7100,172.151.29.48:7100",
      "database.hostname": "172.151.24.238:5433,172.151.23.56:5433,172.151.29.48:5433",
      "database.port": 5433,
      "database.masterhost": "172.151.23.56",
      "database.masterport": "7100",
      "database.user": "yugabyte",
      "database.password": "yugabyte",
      "database.dbname": "cdc_cc3b35",
      "snapshot.mode": "never",
      "admin.operation.timeout.ms": 600000,
      "socket.read.timeout.ms": 300000,
      "max.connector.retries": "10",
      "operation.timeout.ms": 600000,
      "topic.creation.default.compression.type": "lz4",
      "topic.creation.default.cleanup.policy": "delete",
      "topic.creation.default.partitions": 2,
      "topic.creation.default.replication.factor": "1",
      "tasks.max": "5",
      "connector.class": "io.debezium.connector.yugabytedb.YugabyteDBgRPCConnector",
      "database.server.name": "db_cdc",
      "database.streamid": "332df723114dc3ad3d4b5efa82173591",
      "table.include.list": "public.test_cdc_238adc",
      "database.sslrootcert": "/kafka/ca.crt"
    }
  }'


  curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" \
  localhost:8083/connectors/ \
  -d '{
    "name": "ybconnector_cdc_1e628a_test_cdc_f2a058",
    "config": {
      "database.master.addresses": "172.151.29.48:7100,172.151.24.238:7100,172.151.23.56:7100",
      "database.hostname": "172.151.29.48:5433,172.151.24.238:5433,172.151.23.56:5433",
      "database.port": 5433,
      "database.masterhost": "172.151.23.56",
      "database.masterport": "7100",
      "database.user": "yugabyte",
      "database.password": "yugabyte",
      "database.dbname": "cdc_1e628a",
      "snapshot.mode": "never",
      "admin.operation.timeout.ms": 600000,
      "socket.read.timeout.ms": 300000,
      "max.connector.retries": "10",
      "operation.timeout.ms": 600000,
      "topic.creation.default.compression.type": "lz4",
      "topic.creation.default.cleanup.policy": "delete",
      "topic.creation.default.partitions": 2,
      "topic.creation.default.replication.factor": "1",
      "tasks.max": "5",
      "connector.class": "io.debezium.connector.yugabytedb.YugabyteDBgRPCConnector",
      "database.server.name": "db_cdc",
      "database.streamid": "14c46389ace4eb8c584ed8b8254bc3dc",
      "table.include.list": "public.test_cdc_f2a058",
      "database.sslrootcert": "/kafka/ca.crt"
    }
  }'

YugabyteDB version

2024.2.0.0-b61

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.