CoAnSys
CoAnSys copied to clipboard
Avoid frequent HDFS writes in documents similarity
After inspecting several PIG scripts it seems there are plenty of redundant HDFS writes, followed by instant reads which could be optimized.
Apparently this was used for debugging purposes so it will be reasonable to handle debug flag when possible and skip redundant writes/read when flag is not set.
This could significantly improve overall performance.