deltacat icon indicating copy to clipboard operation
deltacat copied to clipboard
verified publisher icon ray-project

[V2][API] Daft reader for DeltaCAT catalogs in general

Open raghumdani opened this issue 1 year ago • 0 comments

This feature allows reading the DeltaCAT catalog (Iceberg, internal catalog etc.) into a Daft distributed dataframe. It also provides hints regarding data layout to Daft reader so that the reads are optimized and no worse than existing compactor.

Expected Features

  • [ ] Reading entire table into memory.
  • [ ] Leverage Partition Pruning using DeltaCAT metadata.
  • [ ] Leverage sort order policy using DeltaCAT metadata.
  • [ ] Support merge on read and hence, delete deltas.
  • [ ] Unblocks #170.

raghumdani avatar Feb 22 '24 00:02 raghumdani