[Feature][Connector-Paimon] Support dynamic bucket splitting improves Paimon writing efficiency
Purpose of this pull request
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
- [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
- [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
- [ ] If you are contributing the connector code, please check that the following files are updated:
- Update change log that in connector document. For more details you can refer to connector-v2
- Update plugin-mapping.properties and add new connector information in it
- Update the pom file of seatunnel-dist
- [ ] Update the
release-note.
cc @dailai and @TaoZex
Please retrigger the ci.
Thinks @hawk9821 . Good job. I think your e2e case needs to be added to the case of multi-parallelism, the current case is all single parallelism. In this way, we can effectively verify whether the dynamic bucketing will change depending on the degree of parallelism of the job. Also, I think you should check the bucket count in every case instead of making a separate case. In addition, each of your cases should verify that the dynamic-bucket.target-row-num argument works as expected.
Why are there so many file changes. Maybe you're having some problems with your git operations. Please reopen a pr which must only has your commits. Then you can link this pr in new pr and close this pr.
Why are there so many file changes. Maybe you're having some problems with your git operations. Please reopen a pr which must only has your commits. Then you can link this pr in new pr and close this pr.
Problems caused by rebase , resolved
Thinks @hawk9821 . Good job. I think your e2e case needs to be added to the case of multi-parallelism, the current case is all single parallelism. In this way, we can effectively verify whether the dynamic bucketing will change depending on the degree of parallelism of the job. Also, I think you should check the bucket count in every case instead of making a separate case. In addition, each of your cases should verify that the dynamic-bucket.target-row-num argument works as expected.
get, add e2e case PaimonSinkDynamicBucketIT.testParallelismBucketCount PaimonSinkDynamicBucketIT.testCDCParallelismBucketCount
LGTM @Hisoka-X PTAL