Epic: Apache DataFusion Integration

[AdamGS]

Vortex Issues

• [ ] #1507
• [ ] Output ordering support (reverting #2429)
• [ ] Respecting more global configurations (or explicitly error if they are used), and making more vortex-specific things configurable.
  • [ ] FileSinkConfig::table_partition_cols and FileSinkConfig::keep_partition_by_columns (Not sure if that's still relevant)
• [ ] Once released in DataFusion, support decimal32/64 conversion to Arrow. https://github.com/vortex-data/vortex/issues/4668
• [ ] Support Dynamic filter expressions https://github.com/vortex-data/vortex/issues/4034
• [ ] Metadata table function - https://github.com/vortex-data/vortex/issues/5144

DataFusion issues we filed

• https://github.com/apache/datafusion/issues/15809
• https://github.com/apache/datafusion/issues/13720 (I think FileMetadataCache might've solved this issue, need to make sure)
• https://github.com/apache/datafusion/issues/13325 (Potentially solved)

DataFusion issues we're interested in

• https://github.com/apache/datafusion/issues/14993

Related

• [ ] If there's willingness, support Vortex in datafusion-dft to have an easy demo https://github.com/datafusion-contrib/datafusion-dft/issues/336

Older solved issues

• [x] Migrating from DataSink to FileSink once the next major release (45) goes out.
• [x] FileScanConfig::limit
• [x] FileScanConfig::table_partition_cols

[haohuaijin]

this https://github.com/vortex-data/vortex/discussions/5481 maybe relate