cp-all-in-one
cp-all-in-one copied to clipboard
confluentinc
Confluent Platform Quick Start (Docker): Datagen Connector not found
Running Docker for windows.
$ docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:23:10 2020
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:29:16 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Windows Version 10.0.18362 Build 18362
I'm working through the Confluent Platform Quick Start (Docker).
Initially, step 2 shows all containers running, but eventually the connect container exits:
$ docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------------------
broker /etc/confluent/docker/run Up 0.0.0.0:9092->9092/tcp
connect /etc/confluent/docker/run Exit 137
control-center /etc/confluent/docker/run Up 0.0.0.0:9021->9021/tcp
ksql-datagen bash -c echo Waiting for K ... Up
ksqldb-cli /bin/sh Up
ksqldb-server /etc/confluent/docker/run Up (healthy) 0.0.0.0:8088->8088/tcp
rest-proxy /etc/confluent/docker/run Up 0.0.0.0:8082->8082/tcp
schema-registry /etc/confluent/docker/run Up 0.0.0.0:8081->8081/tcp
zookeeper /etc/confluent/docker/run Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp
For step 3, the Datagen Connector isn't found. I consulted the referenced Issue: Cannot locate the Datagen Connector, but the resolutions do not produce the desired results.
Upon issuing the following command: docker-compose build --no-cache connect
The output is as follows:
connect uses an image, skipping
Upon issuing the following command: docker-compose logs connect | grep -i Datagen
There is no output.
Upon issuing the following command: docker-compose exec connect ls /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen/lib/
The output is as follows:
ERROR: No container found for connect_1
If I attempt to restart the connect container and issue the above command again before it exits, it does produce the following:
$ docker-compose exec connect ls /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen/lib/
automaton-1.11-8.jar generex-1.0.2.jar jakarta.annotation-api-1.3.5.jar jersey-client-2.30.jar kafka-schema-registry-client-5.5.0.jar
avro-1.9.2.jar guava-18.0.jar jakarta.el-3.0.2.jar jersey-common-2.30.jar kafka-schema-serializer-5.5.0.jar
avro-random-generator-0.3.1.jar hibernate-validator-6.0.17.Final.jar jakarta.el-api-3.0.3.jar jersey-media-jaxb-2.30.jar osgi-resource-locator-1.0.3.jar
classmate-1.3.4.jar jackson-annotations-2.10.2.jar jakarta.inject-2.6.1.jar jersey-server-2.30.jar slf4j-api-1.7.26.jar
common-config-5.5.0.jar jackson-core-2.10.2.jar jakarta.validation-api-2.0.2.jar joda-time-2.9.9.jar snakeyaml-1.24.jar
common-utils-5.5.0.jar jackson-databind-2.10.2.jar jakarta.ws.rs-api-2.1.6.jar kafka-avro-serializer-5.5.0.jar swagger-annotations-1.5.22.jar
commons-compress-1.19.jar jackson-dataformat-yaml-2.10.2.jar jboss-logging-3.3.2.Final.jar kafka-connect-avro-data-5.5.0.jar swagger-core-1.5.3.jar
commons-lang3-3.2.1.jar jackson-datatype-joda-2.10.2.jar jersey-bean-validation-2.30.jar kafka-connect-datagen-0.3.2.jar swagger-models-1.5.3.jar
@derekgreer
-
cp-all-in-onehas been verified on macOS and other operating systems - Confluent Platform is not supported on Windows (even with Docker): https://docs.confluent.io/current/installation/versions-interoperability.html#docker so there may be unexpected behavior. Please switch to a supported OS.
- If you want to still try to troubleshoot in the unsupported Windows OS, check the full output of
docker-compose logs connect?
i use centos 7.5 also get this issue, so what should i do, the machine memory is 4g,not enough?
@RenlySir what error messages are you seeing in your environment? Please use docker-compose logs to troubleshoot
Strange ... I gave up yesterday and decided to just play around with the kafka distro directly. I didn't have Java installed, so I had to install that. This morning, I tried running this again to look at the logs and lo and behold it's now working. Is it possible there is some dependency of the containers for Java to be installed on the host operating system?
I am learning kafka by following this artitle Quick Start for Apache Kafka using Confluent Platform (Docker) https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html#ce-docker-quickstart
But I think i also have some issues using the datagenConnector and I dont know how I can fix it. See information below.
Admin@DCNI-014CS5J-LX MINGW64 /c/Code/Tools/cp-all-in-one/cp-all-in-one (5.5.1-post) $ docker-compose build --no-cache connect connect uses an image, skipping
Admin@DCNI-014CS5J-LX MINGW64 /c/Code/Tools/cp-all-in-one/cp-all-in-one (5.5.1-post) $ docker-compose exec connect ls /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen ls: cannot access C:/Program Files/Git/usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen: No such file or directory
Note: I am also using windows 10.
@azimshaik Correct, Windows is not; the Linux containers are
@abbringas You are missing quotes, otherwise, it assumes a local file path...
docker-compose exec connect "ls /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen"
@derekgreer
Is it possible there is some dependency of the containers for Java to be installed on the host operating system?
Unlikely. Docker itself does not require Java, so therefore the containers have no external reliance on it either.
Connect container will exit because you need to configure Docker with more memory (at least 6GB, as mentioned in the linked docs)
Sounds like you've fixed your issue, though, so this can be closed?