flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[FLINK-34865][pipeline-connector]support sync table and column comments

Open qg-lin opened this issue 1 year ago • 6 comments

Currently we don‘t support sync comments of table or column, so the pipeline sink cannot obtain them. Users must get the comments from the source, it's a bad experience.

qg-lin avatar Jul 19 '24 10:07 qg-lin

resolve some problems from PR https://github.com/apache/flink-cdc/pull/3039, thanks for @lvyanquan and @dufeng1010 and @yuxiqian

qg-lin avatar Jul 19 '24 10:07 qg-lin

@yuxiqian Would you like to help review this PR?

leonardBang avatar Aug 01 '24 05:08 leonardBang

Sure, I'll take it.

yuxiqian avatar Aug 01 '24 05:08 yuxiqian

fix paimon image

melin avatar Aug 09 '24 05:08 melin

fix paimon image

@melin thanks

qg-lin avatar Aug 09 '24 05:08 qg-lin

This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.

github-actions[bot] avatar Oct 09 '24 00:10 github-actions[bot]

This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.

github-actions[bot] avatar Nov 09 '24 00:11 github-actions[bot]

hi,@qg-lin thank you for PR, which is very useful for us. I did some tests after meger and found a problem. If user defines transform in pipeline, the current TransformParser#generateProjectionColumns method will filter out column comment, such as:

source:
  type: mysql
  ....
  include-comments.enabled: true

sink:
  type: paimon
  ....

transform:
  - source-table: testdb.test_table1
    projection: \*, date_format(create_sys_tm,'yyyyMMdd') as pt
    primary-keys: id, pt
    partition-keys: pt
    description: project fields and filter
	
pipeline:
  name: MySQL to Paimon Pipeline
  parallelism: 1

MOBIN-F avatar Nov 20 '24 03:11 MOBIN-F

@qg-lin @MOBIN-F would you like to update this PR?

leonardBang avatar Jan 06 '25 16:01 leonardBang

@qg-lin Would you like to rebase your PR to latest master branch?

leonardBang avatar Jan 14 '25 15:01 leonardBang