librdkafka icon indicating copy to clipboard operation
librdkafka copied to clipboard

[Feature request] Vectored I/O for producer

Open viccpp opened this issue 4 years ago • 6 comments

Please consider adding something like writev() to producer functions.

In my application I have messages that consist of segments (this allows me to use fast pool allocator). Before calling produce() I have to allocate a new buffer for each message and linearize it there. This almost kills all benefits of using memory pool (one-segment messages don't need to be linearized).

viccpp avatar Dec 11 '20 09:12 viccpp

Similar - #2219

viccpp avatar Dec 11 '20 09:12 viccpp

How large is each segment?

edenhill avatar Apr 08 '22 09:04 edenhill

256 bytes

viccpp avatar Apr 11 '22 13:04 viccpp

I've done some performance benchmarks with io vectoring, and it turns out that it is actually cheaper to copy smaller chunks into a larger single memory area than it is to pass around an iovec array, which then also limits the number of messages that can be sent in a single sendmsg() call due to IOV_MAX.

edenhill avatar Apr 11 '22 14:04 edenhill

Which is 1024 these days on most platforms.

Alexander-Shukaev avatar Apr 27 '22 14:04 Alexander-Shukaev

When using iovecs for the payload we also need to pass at least one extra iovec per message for the message headers.

edenhill avatar May 03 '22 07:05 edenhill