[Feature][Manager] Supports the extension of data consumption for different MQs

[ciscozhou]

Description

Supports the extension of data consumption for different MQs.

For example, only the consumption configuration of TubeMQ and Pulsar is currently supported. After the refactor, it supports the rapid expansion of other types of MQ, such as Kafka, RocketMQ, and so on.

Design

1. The table DDL:

-- ----------------------------
-- Table structure for inlong_consume
-- ----------------------------
CREATE TABLE IF NOT EXISTS `inlong_consume`
(
    `id`               int(11)      NOT NULL AUTO_INCREMENT COMMENT 'Incremental primary key',
    `consumer_group`   varchar(256) NOT NULL COMMENT 'Consumer group name',
    `description`      varchar(256)          DEFAULT '' COMMENT 'Inlong consume description',
    `mq_type`          varchar(10)           DEFAULT 'TUBE' COMMENT 'Message queue type, high throughput: TUBE, high consistency: PULSAR',
    `topic`            varchar(256) NOT NULL COMMENT 'Topic that needs to be consumed',
    `inlong_group_id`  varchar(256) NOT NULL COMMENT 'Owning inlong group id',
    `filter_enabled`   int(2)                DEFAULT '0' COMMENT 'Whether to filter consume, 0: not filter, 1: filter',
    `inlong_stream_id` varchar(256)          DEFAULT NULL COMMENT 'Inlong stream ID for consumption, if filter_enable is 1, it cannot empty',
    `ext_params`       text                  DEFAULT NULL COMMENT 'Extended params, will be saved as JSON string, such as queue_module, partition_num',
    `in_charges`       varchar(512) NOT NULL COMMENT 'Name of responsible person, separated by commas',
    `status`           int(4)                DEFAULT '100' COMMENT 'Inlong consume status',
    `is_deleted`       int(11)               DEFAULT '0' COMMENT 'Whether to delete, 0: not deleted, > 0: deleted',
    `creator`          varchar(64)  NOT NULL COMMENT 'Creator name',
    `modifier`         varchar(64)           DEFAULT NULL COMMENT 'Modifier name',
    `create_time`      timestamp    NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Create time',
    `modify_time`      timestamp    NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'Modify time',
    `version`          int(11)      NOT NULL DEFAULT '1' COMMENT 'Version number, which will be incremented by 1 after modification',
    PRIMARY KEY (`id`)
) ENGINE = InnoDB
  DEFAULT CHARSET = utf8mb4 COMMENT ='Inlong consume table';

Are you willing to submit PR?

• [X] Yes, I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct