[EPIC] Parquet filter pushdown

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do. DataFusion offers sophisticated "filter pushdown" optimizations into LogicalPlan::TableScan by passing predicates into TableProvider::scan.

This ticket tracks the work to make use of these predicates in the table provider for parquet files, ParquetFileReader. Much of this work has been completed by the writing of this ticket, but I wanted to try and capture it here to both show how far DataFusion has come as well as how close we are to done

There are three types of predicate pushdown:

• [x] Prune row groups based on statistics (do not fetch or decode any pages)
• [ ] Prune column pages based on page level statistics, skip decode of corresponding positions in other columns
• [ ] Prune row indexes based on Expr predicates, and skip decode of corresponding positions in other columns

Work Items

• [x] https://github.com/apache/arrow-datafusion/issues/3360
• [ ] https://github.com/apache/arrow-datafusion/issues/3457
• [ ] https://github.com/apache/arrow-datafusion/issues/3456
• [ ] https://github.com/apache/arrow-datafusion/issues/847
• [ ] #3463
• [ ] #3464
• [x] https://github.com/apache/arrow-datafusion/pull/2946
• [x] https://github.com/apache/arrow-datafusion/issues/2489 / ParquetRecordBatchStream

Related arrow-rs items:

• [x] https://github.com/apache/arrow-rs/issues/2110
• [x] https://github.com/apache/arrow-rs/pull/2044
• [x] https://github.com/apache/arrow-rs/pull/2115
• [x] https://github.com/apache/arrow-rs/pull/2335
• [x] https://github.com/apache/arrow-rs/pull/2435
• [x] https://github.com/apache/arrow-rs/pull/2467