seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Feature] [InfluxDB Source] add read by chunk

Open 15767714253 opened this issue 9 months ago • 3 comments

Purpose of this pull request

This pull request enhances the InfluxDB source connector by introducing chunked data retrieval. The feature allows the connector to efficiently process large datasets by querying and fetching data in chunks.

Does this PR introduce any user-facing change?

No,The existing functionality of the InfluxDB source connector remains unchanged. Users who upgrade to the version containing this patch will have the additional option to enable chunked data retrieval for performance improvements, but this is not a breaking change, and the feature is opt-in.

How was this patch tested?

I have conducted multiple synchronizations in our company's production environment, with a single table containing more than 300 million data. However, this feature puts a significant load on the resources of InfluxDB itself, which is related to the size of the data read.

Check list

  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
  • [ ] If you are contributing the connector code, please check that the following files are updated:
    1. Update change log that in connector document. For more details you can refer to connector-v2
    2. Update plugin-mapping.properties and add new connector information in it
    3. Update the pom file of seatunnel-dist
  • [ ] Update the release-note.

15767714253 avatar May 07 '24 13:05 15767714253

cc @hailin0 @EricJoy2048

Hisoka-X avatar May 08 '24 02:05 Hisoka-X

https://github.com/apache/seatunnel/issues/4572 Change the document to v2 format reference: https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/source/Mysql.md https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Mysql.md

@15767714253

zhilinli123 avatar May 08 '24 03:05 zhilinli123

Hi @15767714253 , any update for this feature?

Hisoka-X avatar Aug 27 '24 07:08 Hisoka-X