[Improve][DataProxy] Unify the Usage of Message Queue

[dockerzhang]

Description

Unify the Usage of Message Queue

InLong Component

InLong DataProxy

Are you willing to submit PR?

• [ ] Yes, I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[liangyepianzhou]

Good idea! I would like to do this.

[liangyepianzhou]

Motivation

Unify the usages of Zone Message Queue. Now the sink and process for the MQ in the data proxy is chaotic. There is a lot of redundant code here, which makes the code's expansion and maintenance ability very poor.

Modification

We want to extract the standard part of the code. Remove the other implementation such as xxSink and federationSink, and replace them with the zoneSink.

MQ zone

• AbstractZoneClusterProducer
• AbstractZoneSinkContext
• AbstractZoneProducer
• AbstractZoneWorker
• AbstractZoneSink

Implement plan

• implement AbstractClusterZoneProducer and AbstractSinkZoneContext.
• Implement the AbstractZoneProducer.
• implement the AbstractZoneSink and AbstractZoneWork.
• Remove the other implementation such as xxSink and federationSink, and replace them with the zoneSink.
• Add a document about how to extend xxZoneMQ.

[dockerzhang]

@liangyepianzhou lia we do not need federation now, you can only keep zone for multi-MQ clusters.

[liangyepianzhou]

OK, thank you for your feedback. I have modify my proposal to delete other parts except MQ zone.

[liangyepianzhou]

The issue is adding the missing zone sink test.

[liangyepianzhou]

The Doc issue can be found at here.

[liangyepianzhou]

@dockerzhang @healchow The code and documentation for this issue have been completed. In order to better show the final effect and avoid the speed of review affecting the development of the code, the previous small PRs are divided into different commits in a PR. Please help review it when you have time, THX.