storm-kafka and kafka_2.9.2 "0.8.x", "0.9.x" compatibility

[andrewmilkowski]

hello

seeking to run storm-kafka against newer kafka codebase (0.8.x, 0.9.x) codebease in contrib/storm-kafka is compatible with 0.7.x only..

any chance in the near term for a port to the newer kafka ?

thanks!

[asmaier]

This would be very useful indeed.

[jarretraim]

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions:

:provided {:dependencies [[storm "0.9.0-wip15"]

[jasonjckn]

I think 0.7 and 0.8 kafka will be in use for a long time, so it makes sense to maintain two branches of storm-kafka for each respectively. It would be awesome if someone did a PR for this, I personally haven't needed 0.8 yet.

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions: Andrew is referring to kafka version. Yes the latest storm-kafka works with 0.9 storm.

On Tue, Sep 24, 2013 at 5:56 AM, Jarret Raim [email protected]:

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions:

:provided {:dependencies [[storm "0.9.0-wip15"]

— Reply to this email directly or view it on GitHubhttps://github.com/nathanmarz/storm-contrib/issues/38#issuecomment-25000074 .

[asmaier]

See also here https://groups.google.com/forum/#!topic/storm-user/OXQ0a9ppcYM

[dschiavu]

Use this project for Kafka 0.8: https://github.com/wurstmeister/storm-kafka-0.8-plus/

[sujeetyadav]

Hi,

I am trying to integrate storm and kafka but I am facing issue such as I am able to deploy the topology but it is not listening to kafka messages.

Following are the maven dependencies I have used: <groupId>org.apache.storm</groupId> <artifactId>storm-core</artifactId> 0.9.1-incubating <groupId>org.apache.storm</groupId> <artifactId>storm-kafka</artifactId> 0.9.2-incubating <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10</artifactId> 0.8.0 <groupId>org.apache.zookeeper</groupId> <artifactId>zookeeper</artifactId> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> 0.8.2.1

When I run this program it gives following log: 4670 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar... 4676 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar /home/sujeet/Downloads/fieldvantage/TimeSeriesDataMigration/target/TimeSeriesDataMigration-0.0.1-SNAPSHOT.jar to assigned location: /usr/local/storm/data/nimbus/inbox/stormjar-009d6969-b1fc-475a-8e3a-3d0b988fecdd.jar 4691 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /usr/local/storm/data/nimbus/inbox/stormjar-009d6969-b1fc-475a-8e3a-3d0b988fecdd.jar 4691 [main] INFO backtype.storm.StormSubmitter - Submitting topology TestTopology in distributed mode with conf {"topology.workers":1,"topology.message.timeout.secs":600,"topology.debug":false,"kafka.broker.properties":{"bootstrap.servers":"localhost:9092","acks":"1","value.serializer":"org.apache.kafka.common.serialization.StringSerializer","key.serializer":"org.apache.kafka.common.serialization.StringSerializer"}}

But when I am sending messages from Kafka producer it is unable to listen it.

Any help would be appreciable.

Thanks & Regards, Sujeet Yadav

[Dennis1122]

Hi,

I too have the same issuewith kafka storm integration. I am not able to read from kafka topic when running on a single node cluster. Everything works fine while running in local mode. @sujeetyadav Are you able to solve the issue.. ??

Any help will be highly appreciated.

Thanks in advance Dennis