confluent-kafka-go icon indicating copy to clipboard operation
confluent-kafka-go copied to clipboard

Add batch consume method to consumer

Open tburmeister opened this issue 4 years ago • 5 comments

This still needs some work, but wanted to get thoughts on the general idea. Goal here is to add a batch consume method, already available in the C and Python clients, for more efficient consumption of messages. I've included an example of usage, as well as two examples for benchmarking the batch consumer against the current Poll method. In my benchmarks, Saw the following speedups for consuming 1M x 512B messages from a local Kafka server:

poll:
real    0m4.138s
user    0m1.471s
sys    0m0.307s

batch consume:
real    0m0.721s
user    0m0.871s
sys    0m0.208s

There are currently two pretty big limitations to my implementation:

  1. You must turn off auto offsets commits;
  2. You must use the Assign method to manually assign topic-partitions; you can't use the high level Subscribe method.

Basically, the commit and assignment callbacks cause a panic.

tburmeister avatar Nov 20 '19 17:11 tburmeister

@confluentinc It looks like @tburmeister just signed our Contributor License Agreement. :+1:

Always at your service,

clabot

ghost avatar Nov 20 '19 17:11 ghost

@edenhill might be this PR can resolve CGO bottleneck? :)

lexasoft123 avatar Nov 11 '20 18:11 lexasoft123

Is this still in active work? it would be nice to also have this on the Go client

syepes avatar Jan 19 '21 07:01 syepes

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

CLAassistant avatar Apr 01 '22 15:04 CLAassistant

Any updates @tburmeister ?

jflambert avatar Jul 14 '22 18:07 jflambert