parquet-java icon indicating copy to clipboard operation
parquet-java copied to clipboard

Implement async IO for Parquet file reader

Open asfimport opened this issue 3 years ago • 0 comments

ParquetFileReader's implementation has the following flow (simplified) -        - For every column -> Read from storage in 8MB blocks -> Read all uncompressed pages into output queue        - From output queues -> (downstream ) decompression + decoding

This flow is serialized, which means that downstream threads are blocked until the data has been read. Because a large part of the time spent is waiting for data from storage, threads are idle and CPU utilization is really low.

There is no reason why this cannot be made asynchronous and parallel. So 

For Column i -> reading one chunk until end, from storage -> intermediate output queue -> read one uncompressed page until end -> output queue -> (downstream ) decompression + decoding

Note that this can be made completely self contained in ParquetFileReader and downstream implementations like Iceberg and Spark will automatically be able to take advantage without code change as long as the ParquetFileReader apis are not changed. 

In past work with async io  Drill - async page reader , I have seen 2x-3x improvement in reading speed for Parquet files.

Reporter: Parth Chandra / @parthchandra

Related issues:

PRs and other links:

Note: This issue was originally created as PARQUET-2149. Please see the migration documentation for further details.

asfimport avatar May 16 '22 18:05 asfimport