inlong icon indicating copy to clipboard operation
inlong copied to clipboard

[Feature] Add InfluxDB Extract and Load Node for Sort

Open dockerzhang opened this issue 2 years ago • 4 comments

Description

About InfluxDB https://github.com/influxdata/influxdb

Add Apache InfluxDB Extract and Load Node for Sort, and you can refer to: https://inlong.apache.org/docs/next/design_and_concept/how_to_extend_data_node_for_sort/#overview

Use case

No response

Are you willing to submit PR?

  • [ ] Yes, I am willing to submit a PR!

Code of Conduct

dockerzhang avatar Jul 19 '22 13:07 dockerzhang

Hi, could you please assign this task to me? Thanks @dockerzhang

Lu0-Wenfeng avatar Jul 21 '22 18:07 Lu0-Wenfeng

Task

Sort module should support InfluxDB.

InfluxDB is an open-source database for storing and analyzing time sequences. InfluxDB ranked the first place in the latest DB-ENGINES time sequence database ranking. Therefore, it will become more widespread among developers in the near future.

In detail, the tasks includes the following subtasks:

  • Implement the Extract Node for InfluxDB
  • Implement the Load Node for InfluxDB
  • Implement the flink connector for InfluxDB
  • Implement the Extract Node and Load Node in Manager module for InfluxDB

Reference

https://inlong.apache.org/docs/design_and_concept/how_to_extend_data_node_for_sort/

https://inlong.apache.org/zh-CN/docs/design_and_concept/how_to_extend_data_node_for_manager/

Lu0-Wenfeng avatar Aug 01 '22 07:08 Lu0-Wenfeng

Motivation

Sort module supports InfluxDB. InfluxDs is a scalable datastore for metrics, events, and real-time analytics. We should support InfluxDB in sort module.

Design

The design will follow following the document Sort Plugin and Manager Plugin

  1. Extend a new Extract Node for InfluxDB
  2. Extend a new Load Node for InfluxDB
  3. Implement the corresponding flink connectors for InfluxDB
  4. Extend Extract Node and Load Node in manager module for InfluxDB

Modification

Load Node

  1. add the new class InfluxDBLoadNode, which inherits the LoadNode class
  2. add the Load for InfluxDB to JsonSubTypes in LoadNode and Node

Extract Node

  1. add the new class InfluxDBExtractNode, which inherits the ExtractNode class
  2. add the Extract for InfluxDB to JsonSubTypes in ExtractNode and Node

Flink Connector

Creating new file called InfluxDB in inlong-sort/sort-connectors, Adding new classes into the file:

  • InfluxDBTableSink

  • InfluxDBTableSource

  • InfluxDBTableFactory

  • ConfigOptions

  • InfluxDBCatalog

  • InfluxDBCatalogFactory

Manager plugin

Follow the document Manager Plugin to extend the extract node and load node

liangyepianzhou avatar Aug 25 '22 13:08 liangyepianzhou