flink-cdc [FLINK-36517][cdc-connect][paimon] use filterAndCommit API for Avoid commit the same datafile duplicate

https://issues.apache.org/jira/browse/FLINK-35938 problem still persists.

storeMultiCommitter.commit API may cause the same datafile commit twice when job restart from failure.

Oct 12 '24 07:10 beryllw

@lvyanquan @leonardBang PTAL

Oct 12 '24 07:10 beryllw

Could you please assist in reviewing this PR? Thank you. @lvyanquan

Oct 15 '24 02:10 beryllw

I agree that the issue of duplicate commits still exists. Our testing in the case of abnormal failover is relatively lacking, can you try adding corresponding test case for this?

Oct 15 '24 08:10 lvyanquan

I agree that the issue of duplicate commits still exists. Our testing in the case of abnormal failover is relatively lacking, can you try adding corresponding test case for this?

I will try, thanks.

Oct 15 '24 10:10 beryllw

The test fails because the checkpointId is always 1. I'll fix it. https://github.com/apache/flink-cdc/pull/3652/files

Nov 05 '24 04:11 beryllw

@beryllw Would you like to backport this fix to release-3.2 branch?

Nov 11 '24 11:11 leonardBang

OK.

Nov 11 '24 12:11 beryllw