flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard
verified publisher icon apache

[FLINK-34688][cdc-connector] CDC framework split snapshot chunks asynchronously

Open loserwang1024 opened this issue 1 year ago • 3 comments

As shown in https://issues.apache.org/jira/browse/FLINK-34688 : In Mysql CDC, MysqlSnapshotSplitAssigner splits snapshot chunks asynchronously(https://github.com/apache/flink-cdc/pull/931). But CDC framework lacks it. If table is too big to split, the enumerator will be stuck, and checkpoint will be influenced( sometime will checkpoint timeout occurs).

loserwang1024 avatar Aug 06 '24 07:08 loserwang1024

@leonardBang , @ruanhang1993 , @Jiabao-Sun , @GOODBOY008 , CC

loserwang1024 avatar Aug 06 '24 07:08 loserwang1024

Thanks @loserwang1024 for the improvement, @GOODBOY008 would you like to help review this PR when you have time>?

leonardBang avatar Aug 06 '24 07:08 leonardBang

@loserwang1024 Please rebase to master and solove confilcts.

GOODBOY008 avatar Aug 06 '24 11:08 GOODBOY008

@loserwang1024 Please rebase to master and solove confilcts.

@GOODBOY008 done it, and all the tests are passed.

loserwang1024 avatar Oct 25 '24 05:10 loserwang1024

@loserwang1024 Could you kindly rebase your PR to latest master branch to resolve potential conflicts?

leonardBang avatar Nov 20 '24 11:11 leonardBang

@loserwang1024 Could you kindly rebase your PR to latest master branch to resolve potential conflicts?

done it

loserwang1024 avatar Nov 22 '24 05:11 loserwang1024