flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[FLINK-35599] Introduce JDBC pipeline sink connector

Open yuxiqian opened this issue 11 months ago • 9 comments

This closes FLINK-35599 by implementing the long-awaited JDBC pipeline sink connector, largely based on @kissycn's work in #3433.


Compared to Zhou's original PR, some changes have been made to address @lvyanquan's comments in #3433:

  • Supports batch writing for PK tables
    • Added support for TruncateTableEvent and DropTableEvent
  • Added missing SerializationSchemaTest and ITCase
  • Simplified code structure, removed a few redundant classes
  • Rebased with master and resolved conflicts

yuxiqian avatar Feb 10 '25 10:02 yuxiqian

CI passed in my forked repo. Mark it ready for review...

yuxiqian avatar Feb 10 '25 10:02 yuxiqian

Why not integrate with flink-cdc-connector?It has implemented pgsql/orcale/mysql.

Mrart avatar Feb 19 '25 07:02 Mrart

Thanks for @ruanhang1993's kindly review, addressed comments in latest commits.

yuxiqian avatar Mar 26 '25 12:03 yuxiqian

Support postgresql? Many OLAP are compatible with postgresql databases. postgresql is preferred

Writing jdbc is an I/O operation, which requires more resources by increasing the flink concurrency. It is recommended that multiple threads be used to write tables in the same task. Supports setting the number of threads。

melin avatar Mar 27 '25 08:03 melin

Thanks for @melin's suggestion! I've refactored JDBC connector structure mimicking flink-connector-jdbc project like this:

  • flink-cdc-pipeline-connector-jdbc-parent
    • flink-cdc-pipeline-connector-jdbc
    • flink-cdc-pipeline-connector-jdbc-core
    • flink-cdc-pipeline-connector-jdbc-mysql
    • ... (other sinks)

Adding more Jdbc-family sink connectors would be easier.

yuxiqian avatar Mar 27 '25 11:03 yuxiqian

Thanks for @ruanhang1993's kindly review, comments addressed with latest commits.

yuxiqian avatar Mar 28 '25 10:03 yuxiqian

Thanks for @ruanhang1993's kindly review, all comments addressed.

yuxiqian avatar Apr 02 '25 08:04 yuxiqian

When is it scheduled to be released?

lhh-yeah avatar May 07 '25 06:05 lhh-yeah