kafka-python icon indicating copy to clipboard operation
kafka-python copied to clipboard

Published 20 hours ago • verified publisher icon dpkp

Kafka memory error

Open ArtyomIvlev opened this issue 1 year ago • 1 comments

Hello, i want to configurate my consumer with queued.max.messages.kbytes. How i can configure that? But KafkaConsumer have't this config?

ArtyomIvlev avatar May 30 '24 09:05 ArtyomIvlev

Description: I found out that my message consumer starts to consume too much memory when the message lags. Locally I was able to set that 50000 messages completely consumes the message consumer's memory (128Mb) and it just shuts down. How can I fix this situation?

ArtyomIvlev avatar May 30 '24 09:05 ArtyomIvlev

Have you tried tuning fetch_max_bytes ? It defaults to 50MB, which is probably too big if you've only got 128MB to work with.

dpkp avatar Feb 13 '25 20:02 dpkp