Multithreaded consumption kafka

[cxin-william]

Is there a demo of multithreading consumption in Kafka？ The order requirements are not high, just need to improve consumption efficiency on the line。 thanks

[jliunyu]

Hi @cxin-william, thanks for asking.

The first method to improve consumption rate issimply adding more consumers in the same group. However, that approach is more suitable for horizontal scaling where you add new consumers by adding new application nodes (containers, VMs, and even bare metal instances).

If the improvement from the above method is not enough for you, there are many ways to do multi-threaded consumer model please refer to : https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/ (It's for java client, but I think we still can use the method for python client).