kafka-go

kafka-go copied to clipboard

Describe the solution you would like

v0.4.18 added this feature, but it would be nice to be able to disable this new feature.

In our use case, we care a lot about latency (and don't need order to be preserved). So having multiple batches in flight at once is actually really beneficial to us (as it improves latency dramatically to not have to wait for the previous batch to complete). Thus it would be great to have an option to allow multiple batches per partition in flight at once. Alternatively, do you have any suggestions for how to achieve multiple parallel batches with this new version of the Writer?

Thanks!

Supporting documentation

• This comment provides historical context about the architecture.
• https://github.com/segmentio/kafka-go/issues/686#issuecomment-1095597487

Hello @simonrad, thanks for raising this issue.

I am curious whether increasing the number of partitions in your Kafka topic could be an acceptable solution to increasing concurrency. While it is true that we have introduced code to retain ordering of messages in the writer, we maintain an ordering queue per topic/partition, so batches are sent concurrently across partitions. Increasing the number of topics or partitions would effectively reduce contention on the latency of Produce requests.

Let me know if that would work, otherwise we can discuss further how to address this issue.

Thanks for the response @achille-roussel !

That solution (using more partitions) won't work for us because of our partitioning scheme. With only one request in flight at a time, each message has to wait for the previous batch of that partition to complete.

Do you think a good solution would be to use multiple Writers (and cycle through them round-robin)?

I forgot to follow up here, apologies for the delay.

Do you think a good solution would be to use multiple Writers (and cycle through them round-robin)?

Yes, this would be an effective solution in my opinion, you should be able to increase parallelism that way 👍