librdkafka icon indicating copy to clipboard operation
librdkafka copied to clipboard

Feature: Expose partition hashing to producer client

Open shanson7 opened this issue 4 years ago • 0 comments

Description

Re-opening old issue #2046. As mentioned in that issue, we send millions of small messages per second and use a key to partition them. The partitioning being consistent across processes is crucial for the consumer of this data.

In addition to removing the key (which we don't need) we would also like to move to application side batching. For both of these features it would be useful to expose the partitioning function. It seems we could likely do this ourselves by simply getting the number of partitions in the output topic and doing a crc32 on the "key". However, I figured it would be worth reopening this to see if this was something that might be worthwhile to expose.

shanson7 avatar Sep 10 '21 15:09 shanson7