elephant-bird icon indicating copy to clipboard operation
elephant-bird copied to clipboard

Published 20 hours ago • verified publisher icon twitter

InputFormat wrapper doesn't work on CDH4

Open rore opened this issue 11 years ago • 4 comments

Trying to use DeprecatedInputFormatWrapper on CDH4, getting the following exception: java.lang.NoSuchMethodException: org.apache.hadoop.mapreduce.TaskAttemptContext.getCounter(java.lang.String, java.lang.String) at java.lang.Class.getMethod(Class.java:1624) at com.twitter.elephantbird.util.HadoopCompat.(HadoopCompat.java:127)

TaskAttemptContext doesn't contain a getCounter method.

rore avatar May 23 '13 09:05 rore

I tested with standard Apache hadoop. It does have that method: see Line 69

I have to admit DeprecatedInputFormatWrapper was the last thing I tested..

rangadi avatar May 23 '13 22:05 rangadi

CDH4 has the same method too. How are you running your job?

rangadi avatar May 23 '13 22:05 rangadi

I'm building and running on CDH 4.1.2, and it doesn't have the method. You can see here: http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-core/2.0.0-mr1-cdh4.1.2/org/apache/hadoop/mapreduce/TaskAttemptContext.java?av=f

rore avatar May 26 '13 07:05 rore

I've probably should have mentioned I'm using the MRv1 version and not YARN.

rore avatar May 26 '13 08:05 rore