manub
Kafka Embedded dependency problems
Hello,
I'm trying to use the version 0.9.0 because I'm using Spark Streaming 2.0.2 and it seems the most compatible version. But this version has some bugs (Java heap problem...) and I need a newer version.
When I try to use a newer version I get some dependencies errors when I start Kafka embedded. The reason seem that my spark-streaming library use kafka-client 0.10.0.1 and if I try to use a newer version from kafka embedded it uses the version 0.10.2.1. If I add this last dependency the code doesn't work because some functions doesn't exist in this new version from kafka client. Is there any way to resolve this issue? The spark version I use: https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10_2.11/2.0.2 I have tried with many differents versions from kafka embedded and only works te version 0.9.0
23:01:05.805 [main] ERROR o.a.z.server.NIOServerCnxnFactory - Thread Thread[main,5,main] died java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:65) ~[idea_rt.jar:na] Caused by: java.lang.NoSuchMethodError: kafka.server.KafkaServer$.$lessinit$greater$default$2()Lorg/apache/kafka/common/utils/Time; at net.manub.embeddedkafka.EmbeddedKafkaSupport$class.startKafka(EmbeddedKafka.scala:467) ~[scalatest-embedded-kafka_2.11-0.14.0.jar:0.14.0] at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:38) ~[scalatest-embedded-kafka_2.11-0.14.0.jar:0.14.0] at net.manub.embeddedkafka.EmbeddedKafka$.start(EmbeddedKafka.scala:55) ~[scalatest-embedded-kafka_2.11-0.14.0.jar:0.14.0]
Have you tried forking the library and setting the versions to what works for you? If you fork starting from the release you use you could possibly get a matching version.
