flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[pipeline-connector/paimon] Support table specific table.properties configuration

Open nickdelnano opened this issue 7 months ago • 3 comments

This PR adds a new format for defining table properties as specific.table.properties.<database_name>.<table_name>.<property>. This allows defining properties that are specific to a table. The table specific property takes precedence.

Paimon sink supports defining table properties as table.properties.*. They apply to ALL tables. In some cases it is warranted to set table properties based on the table use case. Some examples are compaction settings, Iceberg compatibility settings, etc. This PR allows this.

nickdelnano avatar Aug 06 '25 00:08 nickdelnano

@lvyanquan can you review this change?

nickdelnano avatar Aug 08 '25 18:08 nickdelnano

We can modify the table options through transform, but I agree to provide more configurations in the sink as alternative solutions, refer to https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/core-concept/transform/#specify-table-creation-configuration.

Compared to transform, the current configuration requires too many parameters to be set for situations where many tables need to be parameterized(users usually synchronize a large number of tables in one job). Can we provide a solution for parameter configuration based on regular expressions (Selectors)?

lvyanquan avatar Aug 10 '25 02:08 lvyanquan

This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 60 days if no further activity occurs.

github-actions[bot] avatar Dec 09 '25 00:12 github-actions[bot]