[CDCSDK] [PG Parity] Data Integrity: Data types apart from varchar being emitted as NULL in case of a large transaction

[shamanthchandra-yb]

Jira Link: DB-10863

Description

Please refer slack thread in the JIRA.

PG Parity Source connector:

add connector connector_name='ybconnector_cdc_7ae793_test_cdc_1fed1b' stream_id='rs_cdc_7ae793_1200' db_name='cdc_7ae793' connector_host='172.151.20.75' table_list=['test_cdc_1fed1b'] {'name': 'ybconnector_cdc_7ae793_test_cdc_1fed1b', 'config': {'database.master.addresses': '172.151.17.100:7100,172.151.29.128:7100,172.151.18.145:7100', 'database.hostname': '172.151.17.100:5433,172.151.29.128:5433,172.151.18.145:5433', 'database.port': 5433, 'database.masterhost': '172.151.17.100', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_7ae793', 'snapshot.mode': 'never', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 300000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 2, 'topic.creation.default.replication.factor': '1', 'tasks.max': '10', 'connector.class': 'io.debezium.connector.postgresql.PostgresConnector', 'topic.prefix': 'db_cdc', 'plugin.name': 'pgoutput', 'slot.name': 'rs_cdc_7ae793_1200_from_con', 'publication.autocreate.mode': 'filtered', 'publication.name': 'pn_ybconnector_cdc_7ae793_test_cdc_1fed1b', 'table.include.list': 'public.test_cdc_1fed1b'}}

PG Parity Sink connector:

add connector name='jdbc-sink-test_cdc_1fed1b' {'name': 'jdbc-sink-test_cdc_1fed1b', 'config': {'connector.class': 'io.confluent.connect.jdbc.JdbcSinkConnector', 'tasks.max': '10', 'topics': 'db_cdc.public.test_cdc_1fed1b', 'dialect.name': 'PostgreSqlDatabaseDialect', 'table.name.format': 'test_cdc_1fed1b', 'connection.url': 'jdbc:postgresql://172.151.19.75:5432/postgres?user=postgres&password=postgres', 'transforms': 'unwrap', 'transforms.unwrap.drop.tombstones': 'false', 'auto.create': 'true', 'insert.mode': 'upsert', 'pk.fields': 'k', 'pk.mode': 'record_key', 'delete.enabled': 'true', 'auto.evolve': 'true', 'transforms.unwrap.type': 'io.debezium.connector.postgresql.transforms.yugabytedb.YBExtractNewRecordState'}}

The only change I observe is: connector.class between YB connector and PG connector.

Source connector version

fourpointfour/ybdb-debezium:0.3

Connector configuration

NA

YugabyteDB version

2.23.0.0-b120

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.