Review HDFS file read performance enhancements

Open dlmarion opened this issue 3 years ago • 2 comments

I came across https://issues.apache.org/jira/browse/HADOOP-18287 which references some new APIs:

High Performance Vectored Read API openFile ByteBufferPositionedReadable

There is also https://issues.apache.org/jira/browse/HADOOP-18287 which will enable these new features as they become available.

Jun 13 '22 13:06 dlmarion

2.1 has been developed assuming 3.0 or later. If we do use any new APIs, we have to be careful about ensuring we don't break our minimum compatibility (which we can revisit for the next version, but I think is too late in the dev cycle for 2.1)

Jun 14 '22 05:06 ctubbsii

Agreed, this is an issue for a version after 2.1

Jun 14 '22 11:06 dlmarion