flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[FLINK-38691] [cdc connector mysql] Support for MySQL Transaction Boundary Events in Flink CDC Connector

Open tejanshrana opened this issue 4 months ago • 0 comments

Transaction BEGIN/END events (when provide.transaction.metadata is enabled) are currently treated as unknown and skipped, causing loss of transaction boundaries.

Solution

  • Detect transaction metadata events via key schema name
  • Update binlog offset when processing these events
  • Emit events to downstream consumers if includeTransactionMetadataEvents() is set to true (default = false)

Logs from before the change: 2025-11-04 14:52:28 2025-11-04 14:52:28,827 INFO org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter [] - Meet unknown element SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={transaction_id=4541146d-b988-11f0-87f6-0242ac140006:33, ts_sec=1762267948, file=mysql-bin.000003, pos=13381, gtids=4541146d-b988-11f0-87f6-0242ac140006:1-32, server_id=1}} ConnectRecord{topic='mysql_binlog_source.transaction', kafkaPartition=null, key=Struct{id=4541146d-b988-11f0-87f6-0242ac140006:33}, keySchema=Schema{io.debezium.connector.common.TransactionMetadataKey:STRUCT}, value=Struct{status=BEGIN,id=4541146d-b988-11f0-87f6-0242ac140006:33}, valueSchema=Schema{io.debezium.connector.common.TransactionMetadataValue:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}, just skip.

After the change, the transaction start and end events look like below: {"status":"BEGIN","id":"4541146d-b988-11f0-87f6-0242ac140006:47","event_count":null,"data_collections":null} {"status":"END","id":"4541146d-b988-11f0-87f6-0242ac140006:47","event_count":1,"data_collections":[{"data_collection":"sample.sample","event_count":1}]}

tejanshrana avatar Nov 04 '25 16:11 tejanshrana