Added docker standalone support.

[monksy]

This adds in support so that you can stand up this connector and have it up and working from a local build. Also, this gives support to customize the subreddit, comments subreddit, and broker via environment variable.

Please note: the blank lines in connect-standalone.properties are neccessary because the environment variables are added to the property files and it's overriding the defaults from what you had.

[Guberlo]

Thank you very much, this helps a lot! I'd like to ask you how did you created the topics. When I try to run it, after that I've started two different containers with ZK and Kafka server (everything working), it works but after 2 minutes I get a TimeoutException because no node was assigned.

Should I create a new container like this in order to create a topic?

docker run -e KAFKA_ACTION=create-topic -e KAKFA_SERVER=10.0.100.23 -e KAFKA_TOPIC=$1 --network tap --ip 10.0.100.24 --name kafkaTopic -it tap:kafka

Can you help me?

This is the error that I get:

[2020-04-17 19:23:10,563] INFO Kafka startTimeMs: 1587151390562 (org.apache.kafka.common.utils.AppInfoParser:119) [2020-04-17 19:25:10,566] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:238) org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. [2020-04-17 19:25:10,575] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:238) org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. [2020-04-17 19:25:10,578] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:130) org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties. at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64) at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:83) Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58) ... 2 more Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

Thank you again for your work.

[monksy]

So what i did is that I stood up KA+ZK with docker compose and defined the host IP:

Those containers also create the topics at start.

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    
    environment:
      KAFKA_CREATE_TOPICS: "reddit-posts:3:1,reddit-comments:3:1:compact"
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.244
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

[Guberlo]

Perfect, since I'd like to start my kafka server on this ip 10.0.100.23, I've created this .yaml file.

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    
    environment:
      KAFKA_CREATE_TOPICS: "reddit-posts:3:1,reddit-comments:3:1:compact"
      KAFKA_ADVERTISED_HOST_NAME: 10.0.100.23
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Saved it and typed docker-compose up in that folder. Both Kafka and ZK seems to work fine. Then I moved to the repository folder created the env file as follows:

SUBREDDIT=all
COMMENTS_SUBREDDIT=all
KAFKA_SERVER=localhost:9092

and, after building the image as said, typed the docker run command. The container starts but, after 2 minutes, I get the same error as before.

Is there something that I am missing/doing wrong?

[monksy]

Try setting kafka server to your IP address:9092.

Also what does Docker ps show?

On Mon, Apr 20, 2020 at 8:43 AM Guberlo [email protected] wrote:

Perfect, since I'd like to start my kafka server on this ip 10.0.100.23, I've created this .yaml file.

version: '2' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka ports: - "9092:9092"

environment:
  KAFKA_CREATE_TOPICS: "reddit-posts:3:1,reddit-comments:3:1:compact"
  KAFKA_ADVERTISED_HOST_NAME: 10.0.100.23
  KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
  - /var/run/docker.sock:/var/run/docker.sock

Saved it and typed docker-compose up in that folder. Both Kafka and ZK seems to work fine. Then I moved to the repository folder created the env file as follows:

SUBREDDIT=all COMMENTS_SUBREDDIT=all KAFKA_SERVER=localhost:9092

and, after building the image as said, typed the docker run command. The container starts but, after 2 minutes, I get the same error as before.

Is there something that I am missing/doing wrong?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/C0urante/kafka-connect-reddit/pull/12#issuecomment-616562776, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEWHQ5UU7UYWGDKOHEL3CTRNRGRRANCNFSM4MHPCDBQ .

[Guberlo]

Everything is working now locally with this configuration:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"

    environment:
      KAFKA_CREATE_TOPICS: "reddit-posts:3:1,reddit-comments:3:1:compact"
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.23
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.23:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Thanks for your help. The last thing I'd like to ask you is where is the output stored, or better where should I see it? (Sorry, I'm new to kafka and ZK)

---Edit---

I found the location. Everthing is working now!

[C0urante]

@monksy my sincerest apologies, turns out I forgot to watch my own repo and thus didn't get notified when you filed this PR and the several feature requests! I'll be reviewing this shortly and will get back to you.

[monksy]

Awesome, I appreciate it!

[monksy]

Right now I'm not using the Confluent platform with my kafka setup. (I'm not sure what's community and what's not open source)

I was looking to have this stood up as a self-contained docker container in isolation that'll just run this with whatever broker you have available. I don't want to bind it to a docker compose cluster. I would leave that to the person using it. As far as the images that Confluent provides, I'm not well versed at what they provide. I'm open to suggestions.

[C0urante]

Okay, that helps clarify things. I'm not very familiar either with Confluent's Docker utilities; I think we might be able to just get away with some copy+paste+tweak here with a template file and an invocation of dub (see the links I posted before for examples). I can probably put together something over the weekend if you don't want to bother; LMK

[monksy]

I'm willing to work with you this weekend. I'm not sure how much time I will have between now and Saturday.. But if you want to get together to work through this, I would be up for that.