aiokafka
aiokafka copied to clipboard
Change the implementation of lingering to properly handle retries and force send if batch is full/closed
Currently we send the first message ASAP from the producer. This makes code like this less efficient:
for i in range(10):
await producer.send(topic, msg=str(i).encode())
There are 2 related problems:
- Batches with linger_ms set will only force a longer wait period, but check if next batch is full. We need to change that so that lingering can be skipped if next batch is ready for send.
- If we don't use lingering there is no guarantee on how batches are produced. While this is not a big problem the code above may have non-constant performance. I would still have at least some basic lingering as a default setting to be sure batches are not sent right away if we have a busy loop.