CoAnSys icon indicating copy to clipboard operation
CoAnSys copied to clipboard

Avoid frequent HDFS writes in documents similarity

Open marekhorst opened this issue 8 years ago • 0 comments

After inspecting several PIG scripts it seems there are plenty of redundant HDFS writes, followed by instant reads which could be optimized.

Apparently this was used for debugging purposes so it will be reasonable to handle debug flag when possible and skip redundant writes/read when flag is not set.

This could significantly improve overall performance.

marekhorst avatar Jul 19 '17 14:07 marekhorst