seafowl
seafowl copied to clipboard
Add support for partitioned tables
Currently we don't support these, since we have a custom delta Parquet writer (i.e. plan_to_object_store
) as a way to mitigate memory performance of writes.
This could be achieved in two ways:
- Ditch our custom writing logic and use delta-rs one, which has support for partitioning columns at the expense of getting sub-optimal memory profiles.
- Implement partitioning into our own writing logic.