numaflow
numaflow copied to clipboard
Pause kafka consumer if the buffer is full to avoid topic rebalancing
Summary
If you consume a message from Kafka and it is not able to write to the next buffer because the buffer is full, we will have to tell Kafka that we are going to Pause. Otherwise, Kafka will think this consumer is dead and will trigger a topic rebalance.
NOTE: I haven't validated that this is happening; perhaps we should first validate whether this problem can occur and fix it if need be.
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
has to be implemented before #1519