[QUESTION] Heartbeat thread for CPU intensive consumer: difference to kafka-python

[mparada]

We use kafka-python and have a problem that maybe aiokafka could solve. We have a system that consumes messages one-at-a-time and can take a long time to process (up to 10s of minutes). These are math intensive calculations with a lot of work done by C-libraries (called by pandas, numpy, scipy). Once in a while we get a "heartbeat poll expired" error even though we are still inside the max_poll_interval_ms and our hypothesis is that the processing thread is not allowing the background heartbeat thread to do it's thing. Now my questions:

• Would aiokafka handle better the heartbeat thread for CPU intensive processes? I'm asking because I looked through the code of both libraries and I got the impression that both libraries implement the heartbeat functionality in a similar way.
• Is the async advantage of aiokafka only on the IO side, i.e. communication with the Broker to poll and send messages to topics? Would the performance increase only be apparent if multiple messages are polled together?

BTW Thanks a lot for the awesome work!! 💪 not only here but on kafka-python too!