numaflow icon indicating copy to clipboard operation
numaflow copied to clipboard

Pause kafka consumer if the buffer is full to avoid topic rebalancing

Open vigith opened this issue 11 months ago • 1 comments

Summary

If you consume a message from Kafka and it is not able to write to the next buffer because the buffer is full, we will have to tell Kafka that we are going to Pause. Otherwise, Kafka will think this consumer is dead and will trigger a topic rebalance.

NOTE: I haven't validated that this is happening; perhaps we should first validate whether this problem can occur and fix it if need be.


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

vigith avatar Mar 11 '24 20:03 vigith

has to be implemented before #1519

vigith avatar Jul 29 '24 22:07 vigith