flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[cdc-cli] support recover from a specific savepoint file

Open lvyanquan opened this issue 1 year ago • 3 comments

This closes https://github.com/ververica/flink-cdc-connectors/issues/2940. Use fromSavepoint to be consistent with Flink cli. Some codes are inspired from CliFrontendParser.

lvyanquan avatar Jan 03 '24 12:01 lvyanquan

@PatrickRen PTAL.

lvyanquan avatar Jan 03 '24 13:01 lvyanquan

@lvyanquan 您好,我使用您的PR打了包,结果报错,错误如下 Exception in thread "main" com.ververica.cdc.shaded.com.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: --fromSavepoint at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.handleUnknownToken(DefaultParser.java:561) at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.handleLongOptionWithoutEqual(DefaultParser.java:383) at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.handleLongOption(DefaultParser.java:334) at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.handleToken(DefaultParser.java:540) at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.parse(DefaultParser.java:712) at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.parse(DefaultParser.java:679) at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.parse(DefaultParser.java:660) at com.ververica.cdc.cli.CliFrontend.main(CliFrontend.java:50)

执行的命令 bash /opt/flink-cdc-3.0.0/bin/flink-cdc.sh /opt/flink-cdc-3.0.0/conf/mysql-to-starrocks.yaml --fromSavepoint file:/opt/flink-storage/savepoint/savepoint-f414a3-f2540c5684d0

请问是有什么需要注意的地方吗?

Jazzylol avatar Jan 30 '24 11:01 Jazzylol

@Jazzylol Have you updated lib/flink-cdc-dist-3.0-SNAPSHOT.jar? I have tested again and run correctly.

lvyanquan avatar Jan 31 '24 06:01 lvyanquan

Addressed it.

lvyanquan avatar Mar 02 '24 05:03 lvyanquan

Work like a charm.

PRNDA avatar Mar 04 '24 14:03 PRNDA

Rebased to master.

lvyanquan avatar Mar 07 '24 06:03 lvyanquan

@lvyanquan hi,请问您这个PR,所预期的从savepoint重启Flink CDC Pipeline作业的规范命令是这样吗?

在${FLINK_CDC_HOME}目录下执行:
$ ./bin/flink-cdc.sh mysql-to-doris.yaml -s /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab

因为我没有看到明确的重启命令说明*,所以对此怀有疑惑。并且我认为应当明确指出。

*您引用的链接中,我看到的仍是针对jar作业的重启方式。 $ ./bin/flink run
--detached \ --from-savepoint /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab
./FlinkCDCExample.jar

monologuist avatar May 30 '24 07:05 monologuist

@lvyanquan hi,请问您这个PR,所预期的从savepoint重启Flink CDC Pipeline作业的规范命令是这样吗?

在${FLINK_CDC_HOME}目录下执行:
$ ./bin/flink-cdc.sh mysql-to-doris.yaml -s /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab

因为我没有看到明确的重启命令说明*,所以对此怀有疑惑。并且我认为应当明确指出。

*您引用的链接中,我看到的仍是针对jar作业的重启方式。 $ ./bin/flink run --detached \ --from-savepoint /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab ./FlinkCDCExample.jar

经多次测试,我确认这个命令可行。

monologuist avatar May 31 '24 01:05 monologuist