seafowl icon indicating copy to clipboard operation
seafowl copied to clipboard

Add support for partitioned tables

Open gruuya opened this issue 7 months ago • 0 comments

Currently we don't support these, since we have a custom delta Parquet writer (i.e. plan_to_object_store) as a way to mitigate memory performance of writes.

This could be achieved in two ways:

  1. Ditch our custom writing logic and use delta-rs one, which has support for partitioning columns at the expense of getting sub-optimal memory profiles.
  2. Implement partitioning into our own writing logic.

gruuya avatar Nov 29 '23 14:11 gruuya