confluent-kafka-go
confluent-kafka-go copied to clipboard
Add batch consume method to consumer
This still needs some work, but wanted to get thoughts on the general idea. Goal here is to add a batch consume method, already available in the C and Python clients, for more efficient consumption of messages. I've included an example of usage, as well as two examples for benchmarking the batch consumer against the current Poll
method. In my benchmarks, Saw the following speedups for consuming 1M x 512B messages from a local Kafka server:
poll:
real 0m4.138s
user 0m1.471s
sys 0m0.307s
batch consume:
real 0m0.721s
user 0m0.871s
sys 0m0.208s
There are currently two pretty big limitations to my implementation:
- You must turn off auto offsets commits;
- You must use the
Assign
method to manually assign topic-partitions; you can't use the high levelSubscribe
method.
Basically, the commit and assignment callbacks cause a panic.
@confluentinc It looks like @tburmeister just signed our Contributor License Agreement. :+1:
Always at your service,
clabot
@edenhill might be this PR can resolve CGO bottleneck? :)
Is this still in active work? it would be nice to also have this on the Go client
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.
Any updates @tburmeister ?