inlong icon indicating copy to clipboard operation
inlong copied to clipboard

[Feature][Manager] Add Apache Doris load node management

Open dockerzhang opened this issue 3 years ago • 3 comments

Description

Add Apache Doris Node Management, you can refer to: https://inlong.apache.org/docs/next/design_and_concept/how_to_extend_data_node_for_manager

Dashboard: https://inlong.apache.org/docs/next/design_and_concept/how_to_write_plugin_dashboard

Use case

No response

Are you willing to submit PR?

  • [ ] Yes, I am willing to submit a PR!

Code of Conduct

dockerzhang avatar Jul 14 '22 08:07 dockerzhang

Motivation

To surport the ability to Doris data integration, we need to add Apache Doris Load Node for management

Design

The design mainly follows the document Manager Plugin

  1. Add corresponding SinkType enumeration in enumeration type org.apache.inlong.manager.common.Enums.
  2. In org.apache.inlong.manager.common.Pojo.Sink,create folder path,create the corresponding entity class.
  3. In the org.Apache.Inlong.Manager.Service.Sink path, created under the corresponding tools
  4. Support data source to LoadNode conversion function, reference code org.Apache. Inlong.Manager.Service.Sort.Util.LoadNodeUtils

Implementation

  1. Add DORIS in enumeration type org.apache.inlong.manager.common.enums.SinkType
  2. Create folder "Doris" in org.apache.inlong.manager.common.pojo.sink, and create corresponding entity class:
  • DorisSink
  • DorisSinkDTO
  • DorisSinkRequest
  • DorisColumnInfo
  • DorisTableInfo
  1. Create folder "doris" in org.apache.inlong.manager.service.sink and implement the class:
  • DorisSinkOperator
  1. Add createLoadNode function in org.apache.inlong.manager.service.sort.util.LoadNodeUtils, it is like as follows:
public static DorisLoadNode createLoadNode(DorisSink dorisSink, List<FieldInfo> fieldInfos,
            List<FieldRelation> fieldRelations, Map<String, String> properties){
                 \\TODO 
      }

Loveca avatar Jul 29 '22 06:07 Loveca

@dockerzhang hi, can you assign this to me?

Loveca avatar Jul 29 '22 06:07 Loveca

@Loveca LGTM

dockerzhang avatar Jul 29 '22 14:07 dockerzhang

Duplicate with #6232, close this one.

healchow avatar Oct 26 '22 08:10 healchow