incubator-streampark icon indicating copy to clipboard operation
incubator-streampark copied to clipboard

[SPIP-3][Console] Introduce flink cdc pipeline

Open Mrart opened this issue 1 year ago • 3 comments

Search before asking

  • [X] I had searched in the SPIP and found no similar SPIP.

Motivation

For example, SPIP-2 supports catalog management, and we also support flink cdc, allowing us to submit a flink cdc yaml file to complete the development of synchronous jobs. This has the benefit of fully supporting the table schema change and multi-table synchronization capabilities of flink cdc. Improve the overall product user experience

Design Detail

Part 1: The front-end changes are minimal, adding support for the job type, flink cdc, and adding support for flink cdc yaml in the sql edit box

Part 2: As with the current flink sql logic, yaml is compiled into a jar and submitted to the yarn/k8s cluster 1. app create add flink cdc type 2. createPipelineInstance main class flink cdc main 3. ApplicationActionServiceImp#getUserJarAndAppConf

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

  1. e2e test for flink cdc

Code of Conduct

Mrart avatar Aug 06 '24 02:08 Mrart

https://github.com/apache/incubator-streampark/issues/3857

Mrart avatar Aug 06 '24 04:08 Mrart

Plan

  1. [Feature] Add CDC jar client for support cdc yaml #4163
  2. [Feature] Backend modifications support the cdc yaml api #4164
  3. [Feature] The front-end is modified to support the cdc yaml api #4165
  4. [Feature] E2E to support the cdc yaml api #4165

Mrart avatar Jan 12 '25 06:01 Mrart

2.2 roadmap dont have this feature?

liuyq2step avatar Nov 21 '25 05:11 liuyq2step