Slow message processing speed can be improved with different underlying library

[andydavidson]

Hi,

I love flask_kafka but I have been battling trying to improve the consumer message throughput from the ~30-40 messages per second that I can do on my modern computer.

I tried changing the underlying library from the pure python Kafka library to the librdkafka based confluent_kafka and the message throughput went up from 30-5000 per second with no other changes.

I made a fork and changed the example file to show what change is needed in this library to make it work: https://github.com/andydavidson/flask_kafka

Do you think this is something we can adopt into the mainline flask_kafka project? It's only really the change in configuration, e.g. bootstrap-servers becomes bootstrap.servers that means the change will impact most of the users. :-(

I am not happy with the interrupt behaviour - the consumer sometimes fails to stop when you send the signal with ctrl-c, but perhaps someone with a little more experience can improve this.

[andydavidson]

Here's the comparison to the example and the library

https://github.com/NimzyMaina/flask_kafka/compare/master...andydavidson:master