numaflow icon indicating copy to clipboard operation
numaflow copied to clipboard

Make Kafka source resilient to broker outages

Open nkoppisetty opened this issue 9 months ago • 0 comments

Summary

What change needs making? Change in Kafka client to make it resilient to broker outages.

Use Cases

When would you use this? When a Kafka broker experiences an outage, the Kafka source fails to read incoming messages. It attempts to reconnect to the same broker, causing data loss during the duration of the outage. This issue is common and expected, so it's critical to improve the resilience of the Kafka source to broker outages. By doing so, the Kafka source can continue reading incoming messages during an outage, preventing data loss and ensuring seamless data flow.


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

nkoppisetty avatar May 02 '24 19:05 nkoppisetty