commoncrawl
commoncrawl copied to clipboard
VerifyError
I'm trying Common Crawl w/ Hadoop 0.20.205 and I'm getting the following:
Exception in thread "main" java.lang.VerifyError: (class: org/commoncrawl/hadoop/io/JetS3tARCSource, method: configureImpl signature: (Lorg/apache/hadoop/mapred/JobConf;)V) Incompatible argument to function at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:819) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:864) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:890) at org.commoncrawl.hadoop.io.ARCInputFormat.configure(ARCInputFormat.java:159) at com.digitalpebble.behemoth.io.commoncrawl.CommonCrawlConverterJob.run(CommonCrawlConverterJob.java:54) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at com.digitalpebble.behemoth.io.commoncrawl.CommonCrawlConverterJob.main(CommonCrawlConverterJob.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
I've tried clean builds, etc. Anyone have any ideas? AIUI, this error usually comes from version mismatches. I deleted the jetst 0.6 version in my Hadoop lib directory, but I suspect there is an older version somewhere else in the classpath. Anyone else seen this?
Looks to be an old version of JetS3t packaged with Hadoop:
http://stackoverflow.com/questions/4686036/hadoop-throwing-java-lang-verifyerror-after-exporting-jar-file