numaflow
numaflow copied to clipboard
Make Kafka source resilient to broker outages
Summary
What change needs making? Change in Kafka client to make it resilient to broker outages.
Use Cases
When would you use this? When a Kafka broker experiences an outage, the Kafka source fails to read incoming messages. It attempts to reconnect to the same broker, causing data loss during the duration of the outage. This issue is common and expected, so it's critical to improve the resilience of the Kafka source to broker outages. By doing so, the Kafka source can continue reading incoming messages during an outage, preventing data loss and ensuring seamless data flow.
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.