Multiple webhook brokers

[Abhishekvrshny]

This is more of a question. If I have multiple webhook configurations and multiple instances of pulsar-beam brokers are running, how does the assignment of topics happens? Or, do all brokers try to read from all topics?

[zzzming]

At this moment, we only support one pulsar-beam broker. So the broker just reads webhook configuration and pushes messages. Multiple broker is required for capacity and large volume of messages. I will have some algorithm to balance the load among brokers. Most likely I will hash the topic names and spread them across multiple brokers. So one broker only reads a range of topics and a topic would not be read by multiple brokers. Actually I do have an issue open to track this https://github.com/kafkaesque-io/pulsar-beam/issues/42.

@Abhishekvrshny ^