problem producing with new topic from docker-compose ches/kafka

[dianeswan]

Hello I have been using ches/kafka with docker compose from October last year without problem. Recently I created some new topics but any attempt to produce/consume with those topics fails, although my program still appears to work with the old topics.

In an attempt to investigate I created a new docker-compose.yml with just zookeeper and kafka using new file space. When I run the producer/consumer test I get the same problem, here is some interaction and output; $ docker-compose ps Name Command State Ports

kafka_kafka_1 /start.sh Up 7203/tcp, 0.0.0.0:9092->9092/tcp kafka_zookeeper_1 /docker-entrypoint.sh zkSe ... Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

$ docker-compose run --rm kafka kafka-topics.sh --list --zookeeper zookeeper:2181 Starting kafka_zookeeper_1 ... done

$ docker-compose run --rm kafka kafka-topics.sh --create --topic top1 --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181 Starting kafka_zookeeper_1 ... done Created topic "top1".

$ docker-compose run --rm kafka kafka-topics.sh --list --zookeeper zookeeper:2181 Starting kafka_zookeeper_1 ... done top1

$ docker-compose run --rm kafka kafka-console-producer.sh --topic top1 --broker-list kafka:9092 Starting kafka_zookeeper_1 ... done hello [2018-04-11 14:09:04,661] ERROR Error when sending message to topic top1 with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

If I create zookeeper and kafka from the command line it works

docker run -d --name zookeeper --network test-net zookeeper:3.4 docker run -d --name kafka --network test-net --env ZOOKEEPER_IP=zookeeper ches/kafka docker run --rm --network test-net ches/kafka kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181 docker run --rm --interactive --network test-net ches/kafka kafka-console-producer.sh --topic test --broker-list kafka:9092

    in a separate terminal
    docker run --rm --network test-net ches/kafka kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server kafka:9092

This works fine.

This is my docker-compose file

version: "2" networks: kafka-net:

services: zookeeper: image: zookeeper:3.4 hostname: zookeeper networks:

• kafka-net ports:
• 2181:2181 environment: ZOO_MY_ID: 1 ZOO_PORT: 2181 ZOO_SERVERS: server.1=zookeeper:2888:3888 volumes:
• /home/docker/volumes/test/zookeeper/data:/data
• /home/docker/volumes/test/zookeeper/logs:/log restart: always

kafka: image: ches/kafka hostname: kafka links:

• zookeeper:zookeeper depends_on:
• zookeeper networks:
• kafka-net ports:
• 9092:9092 environment: KAFKA_ADVERTISED_HOST_NAME: kafka ZOOKEEPER_IP: zookeeper KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181" volumes:
• /home/docker/volumes/test/kafka/data:/data
• /home/docker/volumes/test/kafka/logs:/logs restart: always

Sorry to bother you but I have spent a few days on this and not really making any progress.

Thanks Diane

[dianeswan]

I should have added

$ docker --version Docker version 1.12.3, build 6b644ec $ docker-compose --version docker-compose version 1.16.0-rc1, build 9bea70a

OS Ubuntu 14.04