streamnative
[FEATURE] Proxy support for KoP
Hi @jiazhai!
How did you plan to develop this feature? How can I help? :smile:
Currently this is still in our backlog. and not start working on it in our side. If anyone is interested to contribute to this PR welcome to comments on this issue. And Pierre already did some investigation on this, Thanks for @PierreZ's help on this.
@jiazhai @PierreZ is the Envoy proxy able to act like the Pulsar proxy on k8s ?
I mean, in Pulsar we have the "Pulsar proxy" that hides the complexity of the cluster inside k8s.
From the documentation I see that I should setup one Envoy proxy per each Pulsar broker, this is very awkward if you have a large Pulsar cluster, especially in case that the cluster is dynamically scaling up and down.
https://github.com/streamnative/kop/blob/master/docs/envoy-proxy.md
"For each broker with KoP enabled, a dependent Envoy proxy is required. Assuming that you have N brokers whose internal hostname is pulsar-broker-, where i is the broker ID that varies from 0 to N-1."
@eolivelli Yes, not like Pulsar proxy, Envoy acts more like a sidecar. Currently we have no proxy for KoP like Pulsar proxy.
@BewareMyPower @merlimat @jiazhai @PierreZ I have starting working on a Proxy for KOP. I have also sent a PIP to [email protected] in order to add protocol handlers in the Pulsar Proxy.
I will be happy to share my prototype and share my work upstream to this repository.
@eolivelli It would be great to have protocol handlers support in the Pulsar Proxy.
@jiazhai @BewareMyPower @PierreZ @sijie @merlimat I have created this design document for the proxy that I am implementing. https://github.com/streamnative/kop/issues/717
I have already implemented a working prototype, the performances with the proxy are the same as while using regular Pulsar proxy. I have also already implemented support for Token Auth.
I will be happy to contribute the prototype to this repository and continue the development directly here and not in my repo. This way it will be easier to maintain it and also to have more contributions.