[FLINK-34865][pipeline-connector]support sync table and column comments
Currently we don‘t support sync comments of table or column, so the pipeline sink cannot obtain them. Users must get the comments from the source, it's a bad experience.
resolve some problems from PR https://github.com/apache/flink-cdc/pull/3039, thanks for @lvyanquan and @dufeng1010 and @yuxiqian
@yuxiqian Would you like to help review this PR?
Sure, I'll take it.
fix paimon
fix paimon
@melin thanks
This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
hi,@qg-lin thank you for PR, which is very useful for us. I did some tests after meger and found a problem. If user defines transform in pipeline, the current TransformParser#generateProjectionColumns method will filter out column comment, such as:
source:
type: mysql
....
include-comments.enabled: true
sink:
type: paimon
....
transform:
- source-table: testdb.test_table1
projection: \*, date_format(create_sys_tm,'yyyyMMdd') as pt
primary-keys: id, pt
partition-keys: pt
description: project fields and filter
pipeline:
name: MySQL to Paimon Pipeline
parallelism: 1
@qg-lin @MOBIN-F would you like to update this PR?
@qg-lin Would you like to rebase your PR to latest master branch?
