docker-compose fails to bring up Kafka Connect

[natengall]

Tried to bring up Kafka Connect by running docker-compose up on cp-all-in-one-community

It seems to fail with the compose file provided in this repo.

% docker ps          
CONTAINER ID        IMAGE                                         COMMAND                  CREATED              STATUS                             PORTS                                              NAMES
ca00113eda2a        confluentinc/ksqldb-examples:5.5.0            "bash -c 'echo Waiti…"   About a minute ago   Up 51 seconds                                                                         ksql-datagen
3e00003c6873        confluentinc/cp-ksqldb-cli:5.5.0              "/bin/sh"                About a minute ago   Up 51 seconds                                                                         ksqldb-cli
3e82bd1debf3        confluentinc/cp-ksqldb-server:5.5.0           "/etc/confluent/dock…"   About a minute ago   Up 52 seconds (health: starting)   0.0.0.0:8088->8088/tcp                             ksqldb-server
a83c8c044a74        cnfldemos/kafka-connect-datagen:0.3.2-5.5.0   "/etc/confluent/dock…"   About a minute ago   Up 53 seconds (health: starting)   0.0.0.0:8083->8083/tcp, 9092/tcp                   connect
6646e957d46c        confluentinc/cp-kafka-rest:5.5.0              "/etc/confluent/dock…"   About a minute ago   Up 53 seconds                      0.0.0.0:8082->8082/tcp                             rest-proxy
0b27d5828cc5        confluentinc/cp-schema-registry:5.5.0         "/etc/confluent/dock…"   About a minute ago   Up 53 seconds                      0.0.0.0:8081->8081/tcp                             schema-registry
04e1dca419cd        confluentinc/cp-kafka:5.5.0                   "/etc/confluent/dock…"   About a minute ago   Up 53 seconds                      0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp   broker
324792fec66e        confluentinc/cp-zookeeper:5.5.0               "/etc/confluent/dock…"   About a minute ago   Up 54 seconds                      2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp         zookeeper

% docker ps
CONTAINER ID        IMAGE                                   COMMAND                  CREATED              STATUS                        PORTS                                              NAMES
ca00113eda2a        confluentinc/ksqldb-examples:5.5.0      "bash -c 'echo Waiti…"   About a minute ago   Up About a minute                                                                ksql-datagen
3e00003c6873        confluentinc/cp-ksqldb-cli:5.5.0        "/bin/sh"                About a minute ago   Up About a minute                                                                ksqldb-cli
3e82bd1debf3        confluentinc/cp-ksqldb-server:5.5.0     "/etc/confluent/dock…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:8088->8088/tcp                             ksqldb-server
6646e957d46c        confluentinc/cp-kafka-rest:5.5.0        "/etc/confluent/dock…"   About a minute ago   Up About a minute             0.0.0.0:8082->8082/tcp                             rest-proxy
0b27d5828cc5        confluentinc/cp-schema-registry:5.5.0   "/etc/confluent/dock…"   About a minute ago   Up About a minute             0.0.0.0:8081->8081/tcp                             schema-registry
04e1dca419cd        confluentinc/cp-kafka:5.5.0             "/etc/confluent/dock…"   About a minute ago   Up About a minute             0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp   broker
324792fec66e        confluentinc/cp-zookeeper:5.5.0         "/etc/confluent/dock…"   About a minute ago   Up About a minute             2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp         zookeeper

Logs for the connect container isn't showing anything out of the ordinary, but the application on there is crashing and the container is shutting down...

[kmulvey]

same, zookeeper is where dreams go to die

broker             | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
schema-registry    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.18.0.2:2181: No route to host
broker             | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.18.0.2:2181: No route to host
schema-registry    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
broker             | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
schema-registry    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.18.0.2:2181: No route to host
broker             | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.18.0.2:2181: No route to host
schema-registry    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
broker             | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
schema-registry    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.18.0.2:2181: No route to host
broker             | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.18.0.2:2181: No route to host
schema-registry    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
broker             | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
schema-registry    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.18.0.2:2181: No route to host
broker             | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.18.0.2:2181: No route to host

[Poltsi]

This seems to still be the case, both cp-all-in-one-community as well as cp-all-in-one are broken for the same reason. After starting up with: docker-compose up -d The containers all start up, but will die within a minute since they're not able to connect to the zookeeper. The zookeeper however will continue running, and seems to at least be able to open its 2181-port:

[2020-09-11 06:26:33,289] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:os.version=5.8.4-200.fc32.x86_64 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:user.dir=/ (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:os.memory.free=499MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,290] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,292] INFO minSessionTimeout set to 4000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,292] INFO maxSessionTimeout set to 40000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,292] INFO Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /var/lib/zookeeper/log/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-09-11 06:26:33,314] INFO Logging initialized @499ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log)
[2020-09-11 06:26:33,432] WARN o.e.j.s.ServletContextHandler@74650e52{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
[2020-09-11 06:26:33,432] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
[2020-09-11 06:26:33,448] INFO jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 1.8.0_212-b04 (org.eclipse.jetty.server.Server)
[2020-09-11 06:26:33,541] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session)
[2020-09-11 06:26:33,541] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session)
[2020-09-11 06:26:33,543] INFO node0 Scavenging every 660000ms (org.eclipse.jetty.server.session)
[2020-09-11 06:26:33,558] INFO Started o.e.j.s.ServletContextHandler@74650e52{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler)
[2020-09-11 06:26:33,573] INFO Started ServerConnector@31dc339b{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} (org.eclipse.jetty.server.AbstractConnector)
[2020-09-11 06:26:33,574] INFO Started @758ms (org.eclipse.jetty.server.Server)
[2020-09-11 06:26:33,574] INFO Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands (org.apache.zookeeper.server.admin.JettyAdminServer)
[2020-09-11 06:26:33,581] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2020-09-11 06:26:33,585] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-09-11 06:26:33,587] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-09-11 06:26:33,606] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-09-11 06:26:33,608] INFO Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap)
[2020-09-11 06:26:33,614] INFO Snapshotting: 0x0 to /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)

Platform: Linux Fedora 32 + docker 19.3.12 + docker-compose 1.27.1

[ybyzek]

For the containers that are failing (not ZooKeeper), can you please provide logs? docker-compose logs <container>.

[charliee22]

for me, it helped to increase the ram on docker to 8gb

[tl-salvatore-formisano]

I was giving this repo a shot, but all I needed were kafka/zookeeper for some basic local dev needs, and this duo did the trick perfectly:

• https://hub.docker.com/r/bitnami/kafka/
• https://registry.hub.docker.com/r/bitnami/zookeeper

If you have basic needs, you're better off giving these 2 alternatives a shot.

[caueferreira]

I managed to make it work using 2gb ram on docker. I had to run docker-compose up -d a couple of times until I had all services running, after that I had to restart control-center running docker restart control-center.

[dmaksylewicz]

thanks, @charliee22 for the tip, the same increase helped me too

[1ambda]

Connect cluster requires 2GB+ memory, so increasing docker memory to 6GB helped me. The screenshot below is from ctop for cp-all-in-one docker compose components.

[pratap-bhaskar]

@charliee22 Thank you, increasing the memory allowed to start the kafka connect container