Message consumption is slow

[quanghn96]

Steps to reproduce

Consume message from a topic then produce to some other topics

(Scale from 1 to 10 agents)

Expected behavior

Process faster

Actual behavior

Average speed is about 300 message per second.

CPU Usage: 30% RAM Usage: 5%

Versions

• Python version: 3.7
• Faust version: 1.9.0
• Operating system: Ubuntu 18
• Kafka version: 2.4.1

[StephenSorriaux]

Hello,

You can please share a snippet of the code that leads to those results?

[quanghn96]

Here are my example code:

app = faust.App( os.getenv('appname'), broker=os.getenv('broker'), broker_commit_livelock_soft_timeout=os.getenv('livelock'), stream_wait_empty=False )

topic_last=app.topic(os.getenv('topic_name'))

@app.agent(topic_last) async def Last_Topic(requests): async for last in requests.take(max_message, within=max_interval): .....

[Badgerr]

I see the same behavior - very slow reads(dummy.py). I never saw more than 500 messages per second. At the same time another one script(aiokafka_consumer.py) based on AIOKafkaConsumer works at least ten times faster.

dummy.zip

• Python version: 3.6
• Faust version: faust[uvloop,fast,rocksdb]==1.9.0 (the same for the latest version)
• Operating system: RedHat
• Kafka version: 2.2.0

[james-mchugh]

I'm seeing similarly slow behavior. I have an agent that is doing no processing, it is just consuming from a source topic and producing to a sink topic. I have not seen it process at a rate higher than 300 messages per second, and usually it is much lower than that.