inlong
inlong copied to clipboard
[Feature] Add InfluxDB Extract and Load Node for Sort
Description
About InfluxDB https://github.com/influxdata/influxdb
Add Apache InfluxDB Extract and Load Node for Sort, and you can refer to: https://inlong.apache.org/docs/next/design_and_concept/how_to_extend_data_node_for_sort/#overview
Use case
No response
Are you willing to submit PR?
- [ ] Yes, I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Hi, could you please assign this task to me? Thanks @dockerzhang
Task
Sort module should support InfluxDB.
InfluxDB is an open-source database for storing and analyzing time sequences. InfluxDB ranked the first place in the latest DB-ENGINES time sequence database ranking. Therefore, it will become more widespread among developers in the near future.
In detail, the tasks includes the following subtasks:
- Implement the Extract Node for InfluxDB
- Implement the Load Node for InfluxDB
- Implement the flink connector for InfluxDB
- Implement the Extract Node and Load Node in Manager module for InfluxDB
Reference
https://inlong.apache.org/docs/design_and_concept/how_to_extend_data_node_for_sort/
https://inlong.apache.org/zh-CN/docs/design_and_concept/how_to_extend_data_node_for_manager/
Motivation
Sort module supports InfluxDB. InfluxDs is a scalable datastore for metrics, events, and real-time analytics. We should support InfluxDB in sort module.
Design
The design will follow following the document Sort Plugin and Manager Plugin
- Extend a new Extract Node for InfluxDB
- Extend a new Load Node for InfluxDB
- Implement the corresponding flink connectors for InfluxDB
- Extend Extract Node and Load Node in manager module for InfluxDB
Modification
Load Node
- add the new class InfluxDBLoadNode, which inherits the LoadNode class
- add the Load for InfluxDB to JsonSubTypes in LoadNode and Node
Extract Node
- add the new class InfluxDBExtractNode, which inherits the ExtractNode class
- add the Extract for InfluxDB to JsonSubTypes in ExtractNode and Node
Flink Connector
Creating new file called InfluxDB in inlong-sort/sort-connectors, Adding new classes into the file:
-
InfluxDBTableSink
-
InfluxDBTableSource
-
InfluxDBTableFactory
-
ConfigOptions
-
InfluxDBCatalog
-
InfluxDBCatalogFactory
Manager plugin
Follow the document Manager Plugin to extend the extract node and load node