seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Improve][Connector-V2] Improve Paimon source split enumerator

Open xiaochen-zhou opened this issue 9 months ago • 5 comments

Purpose of this pull request Improve Paimon source split enumerator

Does this PR introduce any user-facing change? no

How was this patch tested? exists test

Check list

  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
  • [ ] If you are contributing the connector code, please check that the following files are updated:
    1. Update change log that in connector document. For more details you can refer to connector-v2
    2. Update plugin-mapping.properties and add new connector information in it
    3. Update the pom file of seatunnel-dist
  • [ ] Update the release-note.

xiaochen-zhou avatar Apr 28 '24 00:04 xiaochen-zhou

cc @hailin0

Hisoka-X avatar Apr 28 '24 07:04 Hisoka-X

Please add test case.

dailai avatar Apr 29 '24 01:04 dailai

Please add test case.

Thanks for your review. I think this is an optimization of the existing PaimonSourceSplitEnumerator. Currently, pamion's test case can already cover it.

xiaochen-zhou avatar Apr 29 '24 06:04 xiaochen-zhou

Could you implement stream read for paimon source ? I don't think this is necessary in batch read scenarios.

dailai avatar May 09 '24 01:05 dailai

Could you implement stream read for paimon source ? I don't think this is necessary in batch read scenarios.

Ok, I'd be happy to do it.

xiaochen-zhou avatar May 09 '24 15:05 xiaochen-zhou

@dailai

EricJoy2048 avatar Jul 09 '24 08:07 EricJoy2048