flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

Implement comma-separated parsing for chunk key columns

Open xxntti3n opened this issue 4 months ago • 4 comments

This PR adds support for per-table chunk key column configuration in the PostgreSQL CDC connector, enabling fine-grained control over incremental snapshot chunking. Previously, all tables shared the same chunk key column, which was inefficient for heterogeneous table schemas.

Key Changes

  • ✨ Added chunk key column parsing logic to PostgreSQL DataSourceFactory
  • 🔧 Implemented comma-separated configuration format support

Example Usage

sourceConf: scan.incremental.snapshot.chunk.key-column: public.action_logs:created_at,public.service_logs:created_at

xxntti3n avatar Sep 09 '25 15:09 xxntti3n

Please add more description about this change.

lvyanquan avatar Sep 11 '25 02:09 lvyanquan

@xxntti3n I'm trying to understand what you want to do, do you want to configure a separate chunk key column for schema.table? What's wrong with the previous chunkkeycolumn configuration?

Mrart avatar Sep 19 '25 01:09 Mrart

@xxntti3n I'm trying to understand what you want to do, do you want to configure a separate chunk key column for schema.table? What's wrong with the previous chunkkeycolumn configuration?

yes i want to conf with multiple tables like mysql dialect . Example Conf : scan.incremental.snapshot.chunk.key-column: public.table1:created_at,public.table2:created_at . Currently, Postgres only support 1 table

xxntti3n avatar Sep 25 '25 03:09 xxntti3n

Hi @lvyanquan, @Mrart - I've added the PR description and would appreciate your review. Looking forward to your feedback. Thanks!

xxntti3n avatar Sep 25 '25 08:09 xxntti3n