magpie icon indicating copy to clipboard operation
magpie copied to clipboard

magpie 2.X - Support erasure coding option to terasort

Open chu11 opened this issue 8 years ago • 2 comments

In branch magpie20_erasureencodingterasort, will not work until later

https://issues.apache.org/jira/browse/HDFS-7661

although it's possible to hack teragen/terasort to work around this.

JIRA says may not be fixed for 3.0 release.

chu11 avatar Aug 30 '16 06:08 chu11

also, add to testsuite erasure code option for testing

chu11 avatar Aug 30 '16 06:08 chu11

confirmed

--- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java
@@ -77,7 +77,7 @@ public synchronized void write(Text key,

     public void close(TaskAttemptContext context) throws IOException {
       if (finalSync) {
-        out.hsync();
+          // out.hsync();
       }
       out.close();
     }

Fixes teragen to work for the time being.

chu11 avatar Aug 31 '16 04:08 chu11