Persist data outside of container

[Wassssim]

Hello, I'm wondering how to persist my Kafka queue's data outside of the container, so that if I have to delete the container, I won't lose all my data. I think this can be achieved through volumes, but I have no idea how. can anyone help me?

[prithwi-raj]

@Wassssim you need to set the environment variable explicitly by KAFKA_LOG_DIRS. Otherwise, the default log.dirs uses hostname to construct the log directory name that renews when the container recreates.

environment: KAFKA_LOG_DIRS: "/kafka"

and then volume mount will work as expected

kafka-logdir:/kafka

[Wassssim]

I did that, now I get an Inconsistent Cluster Exception:

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.common.InconsistentClusterIdException: The Cluster ID <ID> doesn't match stored clusterId Some(<another ID>) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong. at kafka.server.KafkaServer.startup(KafkaServer.scala:252) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44) at kafka.Kafka$.main(Kafka.scala:82) at kafka.Kafka.main(Kafka.scala)

[prithwi-raj]

I did that, now I get an Inconsistent Cluster Exception:

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.common.InconsistentClusterIdException: The Cluster ID <ID> doesn't match stored clusterId Some(<another ID>) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong. at kafka.server.KafkaServer.startup(KafkaServer.scala:252) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44) at kafka.Kafka$.main(Kafka.scala:82) at kafka.Kafka.main(Kafka.scala)

Use constant BrokerId (using env KAFKA_BROKER_ID ) if you use a persistent volume.

[ynleborg]

Some error here, setting constant KAFKA_BROKER_ID didn't do anything. Still "The Cluster ID doesn't match stored clusterId Some() in meta.properties." which is true, because after docker-compose down & docker-compose up there is a new clusterID.

According to https://stackoverflow.com/questions/59592518/kafka-broker-doesnt-find-cluster-id-and-creates-new-one-after-docker-restart, since Kafka 2.4 ClusterID is validated.

@Wassssim just add persistent storage also to zookeeper, like this: volumes:

• /home/kouncil/kafka/storage_zoo:/opt/zookeeper-3.4.13/data

[alikhabazian]

Some error here, setting constant KAFKA_BROKER_ID didn't do anything. Still "The Cluster ID doesn't match stored clusterId Some() in meta.properties." which is true, because after docker-compose down & docker-compose up there is a new clusterID.

According to https://stackoverflow.com/questions/59592518/kafka-broker-doesnt-find-cluster-id-and-creates-new-one-after-docker-restart, since Kafka 2.4 ClusterID is validated.

@Wassssim just add persistent storage also to zookeeper, like this: volumes:

• /home/kouncil/kafka/storage_zoo:/opt/zookeeper-3.4.13/data

I used this method and works properly thanks

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
    networks:
      - custom_network
    volumes:
      - ./storage_zoo:/opt/zookeeper-3.4.13/data
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    networks:
      - custom_network

    environment:
      KAFKA_LOG_DIRS: /logs
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./logs:/logs

networks:
  custom_network:
    external: true

[Wassssim]

Some error here, setting constant KAFKA_BROKER_ID didn't do anything. Still "The Cluster ID doesn't match stored clusterId Some() in meta.properties." which is true, because after docker-compose down & docker-compose up there is a new clusterID.

According to https://stackoverflow.com/questions/59592518/kafka-broker-doesnt-find-cluster-id-and-creates-new-one-after-docker-restart, since Kafka 2.4 ClusterID is validated.

@Wassssim just add persistent storage also to zookeeper, like this: volumes:

• /home/kouncil/kafka/storage_zoo:/opt/zookeeper-3.4.13/data

It worked! Thank you!