datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

[EPIC] Update Iceberg integration to use `native_iceberg_compat` scan

Open andygrove opened this issue 4 months ago • 0 comments

What is the problem the feature request solves?

Comet's Iceberg integration currently uses native_comet scan, which does not support complex types.

We are still working on completing the current integration, and then would like to move to using native_iceberg_compat instead for complex type support.

Describe the potential solution

  • Identify design changes needed
  • Implement changes in Comet API
    • ColumnReader
      • PageReadStore ?
      • S3FileIo Support
    • BatchReader
  • Implement corresponding Iceberg changes

Additional context

No response

andygrove avatar Aug 19 '25 14:08 andygrove