seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Feature][Kafka] Support multi-table source read

Open zhilinli123 opened this issue 1 year ago • 10 comments

Purpose of this pull request

Support for kafka multiple data sources

Does this PR introduce any user-facing change?

The same as before

How was this patch tested?

e2e Adds multi-source tests including TOPIC Regex matching

Check list

  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
  • [ ] If you are contributing the connector code, please check that the following files are updated:
    1. Update change log that in connector document. For more details you can refer to connector-v2
    2. Update plugin-mapping.properties and add new connector information in it
    3. Update the pom file of seatunnel-dist
  • [ ] Update the release-note.

zhilinli123 avatar Dec 11 '23 16:12 zhilinli123

issues: https://github.com/apache/seatunnel/issues/5677

zhilinli123 avatar Dec 12 '23 11:12 zhilinli123

PTAL: @hailin0 @Hisoka-X

zhilinli123 avatar Dec 13 '23 11:12 zhilinli123

please resolve conflict

Hisoka-X avatar Dec 14 '23 06:12 Hisoka-X

PTAL:@Hisoka-X @hailin0 thanks !

zhilinli123 avatar Jan 06 '24 07:01 zhilinli123

The topic parameter itself can support the reading of multiple topics. What is the difference?

lightzhao avatar Jan 08 '24 02:01 lightzhao

The topic parameter itself can support the reading of multiple topics. What is the difference?

For example, Topic1 can be consumed from scratch,topic 2 is consumed from the latest offset,topic 3 uses canal_json to parse messages... And so on

zhilinli123 avatar Jan 08 '24 02:01 zhilinli123

@ruanwenjun @TyrantLucifer @hailin0 @Hisoka-X @liugddx @Carl-Zhou-CN PTAL.

EricJoy2048 avatar Jan 12 '24 08:01 EricJoy2048

PTAL: @Hisoka-X @hailin0

zhilinli123 avatar Mar 07 '24 06:03 zhilinli123

PTAL: @Hisoka-X @hailin0

zhilinli123 avatar Mar 15 '24 02:03 zhilinli123

PTAL: @Hisoka-X @hailin0

zhilinli123 avatar May 08 '24 02:05 zhilinli123