delta-rs
delta-rs copied to clipboard
Support non-hive partitioning schema for parquet files
Description
Currently DeltaTablePartition::try_from
only supports hive partitioning. If a parquet path has non-hive parititioning, DeltaTablePartition::try_from
will throw an error. It would be nice to support some non-hive partitioning schemes such as the directory scheme like ./partitioned/2000/part-0.parquet
. User will need to provide the column name when they use this scheme.
Use Case
This feature would make ConvertToDelta operation applicable to non-hive parquet tables.
Related Issue(s)
https://github.com/delta-io/delta-rs/issues/1041, https://github.com/delta-io/delta-rs/issues/1682, https://github.com/delta-io/delta-rs/pull/1686