datafusion-comet
datafusion-comet copied to clipboard
[EPIC] Update Iceberg integration to use `native_iceberg_compat` scan
What is the problem the feature request solves?
Comet's Iceberg integration currently uses native_comet scan, which does not support complex types.
We are still working on completing the current integration, and then would like to move to using native_iceberg_compat instead for complex type support.
Describe the potential solution
- Identify design changes needed
- Implement changes in Comet API
- ColumnReader
- PageReadStore ?
- S3FileIo Support
- BatchReader
- ColumnReader
- Implement corresponding Iceberg changes
Additional context
No response