flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[FLINK-38183] Fix the problem of data loss in the GTID disordered scenario

Open lzshlzsh opened this issue 7 months ago • 3 comments

As the design of https://github.com/apache/flink-cdc/pull/2220,CDC only cares about the maximum GTID position and starts from it. For example, if reading from gtid offset 1-7:9-10, it will automatically adjust to read from 1-10, which causes an error in skipping gitd site 8, thus losing data. In particular, when gtid bit 8 is a large transaction, it will cause more serious data loss. We have encountered this problem many times in the production environment.

More details can be found in [FLINK-38183].

lzshlzsh avatar Aug 03 '25 03:08 lzshlzsh

@ruanhang1993 would you take a look this PR?

leonardBang avatar Aug 04 '25 01:08 leonardBang

This is a duplicate PR for the same issue that I opened and proposed fix awhile ago https://github.com/apache/flink-cdc/pull/3845

cc @leonardBang

mielientiev avatar Aug 05 '25 21:08 mielientiev

Hi @lzshlzsh, this can be closed as https://github.com/apache/flink-cdc/pull/3845 was merged.

lvyanquan avatar Dec 05 '25 09:12 lvyanquan