Help needed to configure application.conf for Confluent distribution of Kafka

[bb56277]

Hi,

I am new to Kafka and have a requirement to read binary log of MySQL and publish it to Kafka. I have followed your "readme" file and have mdoified my.cnf for MySQL.

My application.conf file is as below.

application_conf.txt

My stdout producer works fine and I am able to see the logs in the console. But I am not able to see the same in kafka. I can see that the topics are created.

I run the following command to see view the same from Kafka

./sbt "project runner" "runMain mypipe.runner.KafkaGenericConsoleConsumer kafka_test_test_user_generic localhost:2181 test-consumer-group"

I get the following output. I am not sure how to proceed further. I believe it has to do with my schema registry address. Please kindly help me

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; sup port was removed in 8.0 [info] Loading project definition from /home/kafkauser/confluent-2.0.1/mypipe/project [warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (resolvers) or rename publishing resolver (publishTo). [info] Set current project to mypipe (in build file:/home/kafkauser/confluent-2.0.1/mypipe/) [info] Set current project to runner (in build file:/home/kafkauser/confluent-2.0.1/mypipe/) [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [info] Running mypipe.runner.KafkaGenericConsoleConsumer kafka_test_test_user_generic localhost:2181 test-consumer-group log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:host.name=BIPROSHADOOP2 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:java.version=1.8.0_77 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:java.vendor=Oracle Corporation 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:java.home=/opt/jdk1.8.0_77/jre 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:java.class.path=/home/kafkauser/.sbt/launchers/0.13.9/sbt-launch.jar 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:java.io.tmpdir=/tmp 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:java.compiler=<NA> 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:os.name=Linux 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:os.arch=amd64 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:os.version=3.10.0-327.el7.x86_64 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:user.name=kafkauser 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:user.home=/home/kafkauser 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Client environment:user.dir=/home/kafkauser/confluent-2.0.1/mypipe 16:36:44 INFO [org.apache.zookeeper.ZooKeeper ] Initiating client connection, connectString=localhost:2181 sessionTimeout=400 watcher=org.I0Itec.zkclient.ZkClient@1089e457 16:36:44 INFO [org.apache.zookeeper.ClientCnxn ] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 16:36:44 INFO [org.apache.zookeeper.ClientCnxn ] Socket connection established to localhost/127.0.0.1:2181, initiating session 16:36:44 INFO [org.apache.zookeeper.ClientCnxn ] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1549530daae0006, negotiated timeout = 6000 [WARNING] Avro: Invalid default for field txid: "null" not a ["null",{"type":"fixed","name":"Guid","namespace":"mypipe.avro","size":16}] [WARNING] Avro: Invalid default for field txid: "null" not a ["null",{"type":"fixed","name":"Guid","namespace":"mypipe.avro","size":16}] [WARNING] Avro: Invalid default for field txid: "null" not a ["null",{"type":"fixed","name":"Guid","namespace":"mypipe.avro","size":16}]

[tramchamploo]

First you can check your kafka queue position. Is it Produce and stop or No message at all.

[bb56277]

There is no message

[mardambey]

@bb56277 you mentioned that you see the topic(s) created, is kafka_test_test_user_generic one of them? And to confirm with what @tramchamploo said, the topic is there but has no messages even if you use the kafka console consumer provided with Kafka to try to look at it?

My other question is about the Confluent Platform. Are you using CP1 or CP2? Note that mypipe uses Kafka 0.8, not 0.9 (used by CP2).

[mardambey]

Perhaps of interest here, as of 55f53212bfcd67f5e6054e1aaf7bdfe0f72cdf66 we're adding the ability to easily switch out the Kafka serializer / deserializer in order to allow mypipe to support Confluent's (and other) schema repos.

[wkq72376]

I can not to run mypipe "project runner" "runMain mypipe.runner.KafkaGenericConsoleConsumer $world_$test_generic 127.0.0.1:4180 groupId" project runner need to use which project and groupId is what? thank you very much