cp-docker-images
cp-docker-images copied to clipboard
confluentinc
Kafka Docker - How to increase memory settings
I'm using Confluent Kafka image to run Kafka via docker container on my machine. the kafka broker is often hang and unable to be connected when I did run some tests. It doesn't back to work normally even I killed the container and re-inited it or restarted Docker, etc... I had to use Reset to Factory default , then init the docker container again to make it back to work normally.
It takes my time. I dig into investigating the problem and figured out it may be caused by the memory configuration in the Java process running Kafka
java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xloggc:/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/var/log/kafka -Dlog4j.configuration=file:/etc/kafka/log4j.properties -cp /usr/bin/../share/java/kafka/*:/usr/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/* io.confluent.support.metrics.SupportedKafka /etc/kafka/kafka.properties
It seems 1Gb memory isn't enough and made the kafka in the container hang. I'm looking for a way to increase that configuration to be higher. Anyone knows which ENV var or configuration helps to do that?
KAFKA_HEAP_OPTS - https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L231-L239
KAFKA_HEAP_OPTS- https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L231-L239
how to pass this option to the docker image of the confluent Kafka?
That's the environment variable name, so pass it like all the others
KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"
I passed it and I am getting below error.
===> Launching kafka ...
hostname: Name or service not known
Error: Could not find or load main class "-Xmx512M
Should work, there's an example here that also logs JVM events out to a directory
https://docs.confluent.io/current/installation/docker/development.html#write-garbage-collection-logs-to-an-external-volume
Should work, there's an example here that also logs JVM events out to a directory
https://docs.confluent.io/current/installation/docker/development.html#write-garbage-collection-logs-to-an-external-volume
@cricket007 This works. KAFKA_HEAP_OPTS=-Xmx512M -Xms512M without the double quote. Thanks.