Extend parallel consumer to be able to include multiple consumers/producers of different clusters

[sarwarbhuiyan]

trafficstars

There are a few use cases where a client application may be interested in consuming from multiple clusters and optionally produce to different kafka clusters:

1. In some data aggregation use cases, the data comes from different Kafka clusters (edge or regional clusters) and the data then goes through some transformation before calling some service or going into another Kafka cluster. It might be a neat idea to be able to have support for a list of consumers and list of producers and be able to consume and route the data based on the lambdas

2. In an Active-Passive cluster scenario where have many applications, it is inconvenient to have to re-configure/repoint a consumer after a DR event. As far as the consumer application is concerned, it can handle data being written to either cluster and not care that there's been a DR event. This of course does not work in scenarios where the data is replicated from topics in an Active cluster to a DR cluster.

[astubbs]

Sounds like an idea worth exploring! Draft work will be done here: https://github.com/astubbs/parallel-consumer/compare/features/key-partition-combine...astubbs:features/multi-cluster Community let us know your thoughts!

[astubbs]

Current workaround is to just run multiple PCs. We're thinking on what advantage supporting this inside PC might have..

[johnbyrnejb]

Closing Issue