[proxy] proxy support multi pulsar cluster

[graysonzeng]

Search before asking

• [X] I searched in the issues and found nothing similar.

Motivation

In PIP-121，we support cluster level controlled cluster failover on client side。But I have some issue with this

• when we upgrade pulsar in future，it is more difficult to upgrade client than upgrade proxy。
• client side only support Java，We need to implement other programming language such as go，python。
• cluster level auto failover is great，but i want to implement controlled in topic level。it is more flexible and smaller influences。In this way we can migrate request step by step with topic level，and Observe whether there is an impact on the task associated with the topic。

Solution

• [Idea] We propose an idea，which proxy support multi pulsar cluster，with specified configuration multi cluster。
• [Implement] In this mode，proxy can discover multi pulsar cluster with multi zookeeper connections。when we creating topic，we can specified Mapping between topics and clusters，such as TopicA -> Cluster A, Topic B -> Cluster B, store this metadata in zookeeper。If clients send lookup request to proxy with topicA，and the proxy will respond the specified cluster url (such as Cluster A)。
• [migration] for example，in cluster A we have 10 topics reading and writing., And TopicA -> Cluster A in proxy. I send the swith request(TopicA -> Cluster B) to proxy, the proxy change the mapping metadata in zookeeper, and after then clients request topic A become mapping Cluster B.
• [dependent] we should turn on cluster level geo-replication to ensure the topic data sync between all primary and secondary clusters. Otherwise, it may distribute the topic data into different clusters. And the consumers won’t get the whole data of the topic.

Alternatives

No response

Anything else?

• current architecture situation :

• after support multi pulsar cluster architecture :

Are you willing to submit a PR?

• [X] I'm willing to submit a PR!

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.