iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard

Tracking Issue: Integration with Datafusion

Open marvinlanhenke opened this issue 1 year ago • 1 comments

Motivation:

Once we merge #324 and complete the basic groundwork, the integration crate can be continuously refined. Due to for example more supported features like predicate pushdown, etc.

This list is supposed to evolve over time, so feel free to add missing todos.

Todos:

  • [ ] proper integration tests for table scan (requires us to setup a table with actual snapshots / data )
  • [ ] a cache for IcebergCatalogProvider and IcebergSchemaProvider / so data does not become stale
  • [ ] improve impl ExecutionPlan for IcebergTableScan (once we support filter pushdown, etc.)

marvinlanhenke avatar Apr 28 '24 13:04 marvinlanhenke

https://crates.io/crates/object_store_opendal is available to help integration with object_store.

Xuanwo avatar Jun 21 '24 14:06 Xuanwo