inlong icon indicating copy to clipboard operation
inlong copied to clipboard

[Feature][Manager] Support Kafka

Open dockerzhang opened this issue 3 years ago • 2 comments

Description

Manage Kafka like Pulsar/TubeMQ

Use case

No response

Are you willing to submit PR?

  • [ ] Yes, I am willing to submit a PR!

Code of Conduct

dockerzhang avatar Jul 11 '22 09:07 dockerzhang

@dockerzhang Hi, I want to do this task.

jun0315 avatar Jul 21 '22 10:07 jun0315

Motivation

Manage module supports Apache Kafka. Apache Kafka is an open-source distributed event streaming platform . We should support Apache Kafka in manage module.

Design

The design will follow following the document Manager Plugin

The main modules of Kafka include topic and partition.

image

The mapping relationship between inlong and Kafka is: Kafka's topic is group+stream, and the specific rule is: topic=group_ with_ stream, where with is the separator. The rules can be configured and perceived by users.

Modification

Creating new file called kafka inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/group, Adding new classes into the file:

InlongKafkaInfo.java

InlongKafkaDTO.java

InlongKafkaRequest.java

Creating new file called kafka inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue, Adding new classes into the file:

KafkaOperator.java

KafkaResourceOperator.java

KafkaUtils.java

Manager plugin

Follow the document Manager Plugin.

jun0315 avatar Aug 04 '22 14:08 jun0315