ISSUE-64. Hadoop 3.0 NN drops replicas silently.

[jojochuang]

trafficstars

Fixes #64 This PR changes blockMap of dyno DN from hash map to tree map. The fix was verified internally at Cloudera. Additional stress test will be needed to understand the performance impact of this change. But it is functionally correct for Hadoop 3.0/CDH5.7.

[xkrogen]

Great find, thanks @jojochuang ! Sorry for the delay in my response.

Two questions:

• Why is the comparator a Comparator<Object> instead of Comparator<Block>?
• Will this have any performance implications (on the NameNode) on pre-3.0 versions? I don't want this change to accidentally affect the performance of versions which do not have HDFS-9260. I suspect it will not, but would like your opinion before merging.

[jojochuang]

Hi Eric, Apologies -- my github communication goes into my personal email account and I didn't see this one.

Why is the comparator a Comparator<Object> instead of Comparator<Block>?

Good point. It should be a Block. The code was based on the HDFS-9260.

Performance -- we've not compared the performance yet since our focus is on CDH, and CDH clusters won't even run for long without this patch. I thought about using FoldedTreeSet (implemented in HDFS-9260) but then opted fora simpler, more well known data structure.

[xkrogen]

I'm hesitant to merge this given that the consensus reached on HDFS-13671: Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet regarding the changes in HDFS-9260 seems to be to revert it. I think that this at least needs to be an optionally-enabled feature rather than always-on