aiokafka icon indicating copy to clipboard operation
aiokafka copied to clipboard

Change the implementation of lingering to properly handle retries and force send if batch is full/closed

Open tvoinarovskyi opened this issue 7 years ago • 0 comments

Currently we send the first message ASAP from the producer. This makes code like this less efficient:

for i in range(10):
    await producer.send(topic, msg=str(i).encode())

There are 2 related problems:

  • Batches with linger_ms set will only force a longer wait period, but check if next batch is full. We need to change that so that lingering can be skipped if next batch is ready for send.
  • If we don't use lingering there is no guarantee on how batches are produced. While this is not a big problem the code above may have non-constant performance. I would still have at least some basic lingering as a default setting to be sure batches are not sent right away if we have a busy loop.

tvoinarovskyi avatar Nov 17 '18 16:11 tvoinarovskyi