Multiple kafka clusters

[DamianZaremba]

Currently it's possible to specify multiple brokers for a single cluster.

In order to provider cluster level redundancy, it would be nice to be able to 'dual publish', so the exact same messages are sent 2 (or really N) times.

I've not looked deeply into the existing code to see how easy this would be, but would like to gather some feedback on if others have this use case and thus would a pull request be accepted for it.

I imagine something like;

kafka:
  brokers:
    cluster1:
      - xxxx
      - yyyy
    cluster2:
      - xxxx
      - yyyy

Where cluster1/cluster2 are just identifiers for logging/stats or such.

The main unknown is what the current behaviour in the cluster being down is; if it fails safe at publish time this is probably rather easier, if it assumes a hard dep then likely harder.

[TimEvens]

This would be good but it’ll increase resource usage a bit.

[psikala]

How about deploying two BMP -stations? 1. push to cluster one, and 2. to cluster two.

[allamiro]

I'm not sure about this but I think its more like environment specific and you may need to follow Kafka guidelines on how to setup cluster failover.

https://docs.confluent.io/5.5.0/multi-dc-deployments/replicator/replicator-failover.html

But definitely it would be good to have with the overall setup