seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Improve][Connector-V2][Kafka] Support extract topic from SeaTunnelRow field

Open TaoZex opened this issue 2 years ago • 3 comments

Purpose of this pull request

Support extract topic from SeaTunnelRow field

For test :

source config : 82fc9f9a865e6e735e84f333b9d7f5e

sink config : 33a8b5602541f7a7c3cce8d612a75ae

send records to topic: 78c1df512b3b569634e89398aa0d7ad

Check list

  • [ ] Code changed are covered with tests, or it does not need tests for reason:
  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
  • [ ] If you are contributing the connector code, please check that the following files are updated:
    1. Update change log that in connector document. For more details you can refer to connector-v2
    2. Update plugin-mapping.properties and add new connector information in it
    3. Update the pom file of seatunnel-dist

TaoZex avatar Dec 16 '22 12:12 TaoZex

Add e2e testcase

Writing e2e test for this feature is not easy because the kafka sink connector send records without callback function, so I took a screenshot to prove that the feature is available.

TaoZex avatar Dec 17 '22 05:12 TaoZex

Add e2e testcase

Writing e2e test for this feature is not easy because the kafka sink connector send records without callback function, so I took a screenshot to prove that the feature is available.

  1. Generate fixed data in the data source
  2. Use kafka sink to write topic
  3. Check the message of the topic

hailin0 avatar Dec 17 '22 06:12 hailin0

Add e2e testcase

Writing e2e test for this feature is not easy because the kafka sink connector send records without callback function, so I took a screenshot to prove that the feature is available.

  1. Generate fixed data in the data source
  2. Use kafka sink to write topic
  3. Check the message of the topic

Thanks for your advice.

TaoZex avatar Dec 17 '22 10:12 TaoZex

I added e2e test for this feature and refactor the kafkaIT code.PTAL @hailin0 @EricJoy2048 @Hisoka-X

TaoZex avatar Jan 18 '23 09:01 TaoZex

Let's waiting CICD.

TyrantLucifer avatar Jan 20 '23 05:01 TyrantLucifer