librdkafka icon indicating copy to clipboard operation
librdkafka copied to clipboard

Add new queued.max.messages.total.kbytes option

Open leeavital opened this issue 6 months ago • 1 comments

This option should behave exactly as queued.max.messages.kbytes behaves except it takes into account the queued size of all topic-partition queues at once instead of imposing a per-queue limit.

The issue with a per-queue limit, is that with dynamic partition assignment the number of partitions assigned to a single worker can change dramatically. Thus saying "2000kb" queued could be between 2000kb or ~1.5 gigabytes for a 720 partition topic depending on how many partitions are assigned in an instant.

This is particularly an issue for messages which are well compressed, because the size constraint in both instances is imposed before decompression.

leeavital avatar May 12 '25 20:05 leeavital

Please sign the Contributor License Agreement here before this PR can be approved.
:x: leeavital
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.