Data-Ingestion-Platform
Data-Ingestion-Platform copied to clipboard
XavientInformationSystems
storm jar uber-dataingest-1.0.0.jar com.xavient.storm.dataingest.DataIngestionTopology
Hi team/Sir,
I am deploying the jar as below from the path as /usr/hdp/2.6.4.0-91/storm/lib storm jar uber-dataingest-1.0.0.jar com.xavient.storm.dataingest.DataIngestionTopology and getting error as storm.dependency.jars= -Dstorm.dependency.artifacts={} com.xavient.storm.dataingest.DataIngestionTopology Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/topology/IRichSpout at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: backtype.storm.topology.IRichSpout at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more
My sandbox version is Sandbox information: Created on: 01_02_2018_10_47_41 Hadoop stack version: Hadoop 2.7.3.2.6.4.0-91 Ambari Version: 2.6.1.0-143 Ambari Hash: 2989989d67edacff7e9db702b4cf0c080556dddc Ambari build: Release : 143 Java version: 1.8.0_161 OS Version: CentOS release 6.9 (Final)
The storm version is as below :- 1.1.0
Could Anyone please help me here ,
Thanks Prosenjit
Hi , Thanks for writing me back ,However I have resolved the issue and I have one question ,after I submit the jar then I need to do Open the UI for the application by visiting the URL "http://tomcat-server:port/DataIngestGUI/UI.jsp",I am not finding the UI.jsp file ,could you please help me in this ,
One question If I directly put the csv's JSON's and XML directly into the KAFKA_TOPIC, will the storm spout process it to the next bolt,Kindly let me know.
Thanks Prosenjit
Hi ,
I have resolved the issue however when I pass messages to Kafka_topic like bin/kafka-console-producer.sh --broker-list sandbox-hdp.hortonworks.com:6667 --topic kafka_topic OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N {"id":"XIND10000","author":"Jack White","title":"Hadoop Made Easy","genre":"Programming","price":"50000","publish_date":"2001-09-10","description":"ebook"} {"id":"XIND10002","author":"Steven Berg","title":"Apache Source","genre":"Programming","price":"45000","publish_date":"2001-10-10","description":"ebook"}
and I receive the messages in consumer as well
but when I see them in storm topology ,I get the below error
ava.lang.NullPointerException: topic cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226) at org.apache.storm.kafka.DynamicBrokersReader.
Could you please help me with this ,waiting for your kind reply ,if you want we can have a session remote as well.
Thanks Prosenjit
Hi ,
As agreed can we please look into it , had resolved the above error
but getting the below error java.lang.ExceptionInInitializerError at kafka.metrics.KafkaMetricsGroup$class.newTimer(KafkaMetricsGroup.scala:85) at kafka.consumer.FetchRequestAndResponseMetrics.newTimer(FetchRequestAndResponseStats.scala:26) at kafka.consumer.FetchRequestAndResponseMetrics.<init>(FetchRequestAndResponseStats.scala:35) at kafka.consumer.FetchRequestAndResponseStats.<init>(FetchRequestAndResponseStats.scala:47) at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:60) at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:60) at kafka.utils.Pool.getAndMaybePut(Pool.scala:59) at kafka.consumer.FetchRequestAndResponseStatsRegistry$.getFetchRequestAndResponseStats(FetchRequestAndResponseStats.scala:64) at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:44) at kafka.javaapi.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:34) at org.apache.storm.kafka.DynamicPartitionConnections.register(DynamicPartitionConnections.java:60) at org.apache.storm.kafka.PartitionManager.<init>(PartitionManager.java:70) at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129) at org.apache.storm.daemon.executor$fn__10182$fn__10197$fn__10228.invoke(executor.clj:647) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) at java.lang.Runtime.addShutdownHook(Runtime.java:211) at com.yammer.metrics.Metrics.<clinit>(Metrics.java:21) ... 19 more
Can you please help today is tuesday.
Thanks Prosenjit
This seems to be versioning issue. As you are using newer version of HDP please change the versions of storm, Kafka etc in pom file. Rebuild and run.
@mohnkhan @BigData-Developer @rskumar @nsabharwal I am changed the POM.xml as per the version I am using in Ambari ,please find the pom.xml and the screenshot with the component versions attached .
I am attaching both kindly let me know where I am going wrong.
KIndly help me here .
Thanks Prosenjit Components Version as defined in Ambari Services.docx pom.txt