librdkafka
librdkafka copied to clipboard
Add new queued.max.messages.total.kbytes option
This option should behave exactly as queued.max.messages.kbytes behaves except it takes into account the queued size of all topic-partition queues at once instead of imposing a per-queue limit.
The issue with a per-queue limit, is that with dynamic partition assignment the number of partitions assigned to a single worker can change dramatically. Thus saying "2000kb" queued could be between 2000kb or ~1.5 gigabytes for a 720 partition topic depending on how many partitions are assigned in an instant.
This is particularly an issue for messages which are well compressed, because the size constraint in both instances is imposed before decompression.
Please sign the Contributor License Agreement here before this PR can be approved.
:x: leeavital
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.