Mobius icon indicating copy to clipboard operation
Mobius copied to clipboard

Custom Input Partitioning

Open kngine-rash opened this issue 7 years ago • 0 comments

I'm trying to use newAPIHadoopFile to implement custom InputFormat and RecordReader classes instead of using the default ones provided by Spark. I can do the same in Java but am unable to find the correct way of achieving this using Mobius.

For reference, these are the classes and interfaces I want to override

java.lang.Object
org.apache.hadoop.mapreduce.InputFormat<K,V>
org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

and Interface RecordReader<K,V>

https://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/mapreduce/RecordReader.html

https://hadoop.apache.org/docs/stable/api/index.htmlorg/apache/hadoop/mapreduce/lib/input/FileInputFormat.html

In Mobius, trying to use a custom InputFormat class throws a 'ClassNotFound' exception. I'm guessing it only excepts classes provided by Hadoop.

Does Mobius support custom partitioning ?

kngine-rash avatar Apr 10 '18 18:04 kngine-rash