apache
HadoopInputFile to pass down FileStatus when opening file.
In the current version of the HadoopInputFile implementation:
When performing a newStream, the reference to the FileStatus is lost, which has already been previously consulted to create this class. This means that when you go to the implementation of each FileSystem, it will surely have to be requested again, since you have requested the reference of whether the file exists, when the file is weighed or relevant information to be able to open the file.
Hadoop's openFile() builder API does support this, but it is not on older releases, so until Parquet moves to Hadoop 3.2.0+ only it cannot use the API. And because its a complex and extensible design, it's very hard to use reflection.
HADOOP-19131 adds reflection-friendly entry points for this and other operations, so for releases with the new class, Parquet can pick up the speedup.
Reporter: Oliver Caballero Alvarez
Related issues:
- Improve Parquet IO Performance within cloud datalakes (blocks)
- WrappedIO to export modern filesystem/statistics APIs in a reflection friendly form (depends upon)
- Reduce the number of accesses to S3 object storage (depends upon)
- Include FileStatus when opening a file from FileSystem (is depended upon by)
Note: This issue was originally created as PARQUET-2493. Please see the migration documentation for further details.
