Consumer performance benchmarking

[poonkothaip]

Description

We are using this library based consumer (basic one) to consume messages from MSK. performance wise, this doesnt meet our defined requirements. Can anyone suggest any pointers on how to optimise the parameters or tune to get best performance. to process 5k messages, as of now it takes around 24-45 minutes but our requirement is less than 5 min. Properties used are as below

How to reproduce

NA

Checklist

Please provide the following information:

• [ ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 2.6.0 library and MSK
• [ ] Apache Kafka broker version: 0.1.0
• [ ] Client configuration: {...}-ECS Fargate service
• [ ] Operating system: Linux
• [ ] Provide client logs (with 'debug': '..' as necessary) NA
• [ ] Provide broker log excerpts NA
• [ ] Critical issue Performance

[pranavrth]

The performance is pretty bad. Can you check which part of the code is taking much time? Are you performing some IO operation between consuming messages?

Are you sure that you are using Apache Kafka 0.1.0 version? 0.7.0 was release in 2012.

[poonkothaip]

Hi Pranav Thanks for your note Apache kafka version - 2.7.2 after consuming we do small log operation alone for now. But sometimes it takes upto 1 hr. we have 3 partitions.

Thanks Poongkothai

[pranavrth]

Can you log time for each step? Getting message from the python client should not be taking alot of time.

[poonkothaip]

Sure Pranav. Will do and revert but might take sometime Do we have any benchmarking of this with any other python libraries for kafka. am using the latest confluent-kafka python version anyways

Thanks

[poonkothaip]

Hi I triggered load of 500 messages and have basic consumer code to print the message in both library based version of code Logs show for kafka-python, it took less than a minute to process all 500 (started and done at 12.55). Confluent-kafka took 9 mins (started at 12.43 and ended at 12.52) Unfortunately am not able to attach logs here. But its basic code with just log message inside doing nothing. I suspect consumer poll is taking more time in confluent kafka than kafka python.

Can you share if any bench marking was done earlier

Thanks

[poonkothaip]

Hi @pranavrth - By default do we need to have any threading concept added to improve performance . I run as single threaded application in both cases (kafka-python and confluent -kafka) and no of partitions are more than 1. Ideally threads may be equal to no of partitions in case sequence needs to be maintained. Any suggestions to further improve the performance for confluent kafka based consumer. Also how to overcome consumer poll taking more time in confluent kafka. Thanks

[MSeal]

@poonkothaip apologies this was left unanswered for a long while. The underlying queues are already threaded, you threading above likely won't help unless you have inbound network or disk delays in your loop that hold the CPU for extended time periods. You may need to post code related if you are still seeing this performance with current versions. What's the polling call timeout/when are you polling? Are you doing other operations that are holding up consume calls? What is your consumer config? Additionally logs help a lot if they can be shared to diagnose where something may be going wrong.

[MSeal]

Closing for now. We did a lot of performance improvements and load testing since I posted before. Please open a new issue if there's a performance regression and provide logs / setup if possible. Thanks