flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[pipeline-connector][paimon] add paimon pipeline data sink connector.

Open lvyanquan opened this issue 1 year ago • 5 comments

This close https://github.com/ververica/flink-cdc-connectors/issues/2856. Some codes are inspired by FlinkCdcMultiTableSink in Paimon repo, and add a sinkV2 implement.

lvyanquan avatar Dec 22 '23 07:12 lvyanquan

@PatrickRen PTAL.

lvyanquan avatar Dec 28 '23 09:12 lvyanquan

image paimon在schemaschemachangevent事件产生时会从catalog里加载最新的schema,这个时候schema有可能没被修改,导致写入的数据还是修改ddl之前的schema字段数据,新的字段数据读不出来或者删除字段后出现新的问题 image 是否可以在releasestream之后发送schemaChangeEvebt事件,这样下游获取schema就一定是最新的

yanghuaiGit avatar Jan 29 '24 07:01 yanghuaiGit

Thanks @yanghuaiGit for pointing out this, address it.

lvyanquan avatar Jan 29 '24 10:01 lvyanquan

image com.ververica.cdc.connectors.paimon.sink.PaimonMetadataApplier 静态字段catalog,在反序列化之后,获取的对象里catalog为null,导致com.ververica.cdc.connectors.paimon.sink.PaimonMetadataApplier#applySchemaChange方法执行时为空指针。

catalog应改为 private transient Catalog catalog;,在applySchemaChange时判断是否为空来构建一个catalog image

yanghuaiGit avatar Jan 30 '24 09:01 yanghuaiGit

address it.

lvyanquan avatar Jan 30 '24 09:01 lvyanquan

Support reading data from multiple table messages written to the same topic?

https://github.com/apache/flink-cdc/pull/2938#issuecomment-1970940065

melin avatar Mar 13 '24 06:03 melin

Can multiple table cdc message be written to the same topic?

You can do this by using route in pipeline.

lvyanquan avatar Mar 13 '24 06:03 lvyanquan

paimon latest version is 0.7,we should update paimon version from 0.6 to 0.7

yanghuaiGit avatar Mar 28 '24 03:03 yanghuaiGit

paimon latest version is 0.7,we should update paimon version from 0.6 to 0.7

@lvyanquan Could you take a look at this one? I prefer to catch up with the latest version as well. Also could you rebase the latest master? Thanks

PatrickRen avatar Mar 28 '24 09:03 PatrickRen

paimon latest version is 0.7,we should update paimon version from 0.6 to 0.7

Done and rebased to master.

lvyanquan avatar Apr 01 '24 04:04 lvyanquan

Does kafka header set constant values? For example, if data from multiple data centers is written to the same kafka topic, add a region key to the kafka header.

melin avatar Apr 10 '24 06:04 melin

Thanks @yuxiqian for those comments, I've addressed it and resubmitted.

lvyanquan avatar Apr 15 '24 01:04 lvyanquan