delta-rs icon indicating copy to clipboard operation
delta-rs copied to clipboard

Support non-hive partitioning schema for parquet files

Open junjunjd opened this issue 7 months ago • 0 comments

Description

Currently DeltaTablePartition::try_from only supports hive partitioning. If a parquet path has non-hive parititioning, DeltaTablePartition::try_from will throw an error. It would be nice to support some non-hive partitioning schemes such as the directory scheme like ./partitioned/2000/part-0.parquet. User will need to provide the column name when they use this scheme. Use Case This feature would make ConvertToDelta operation applicable to non-hive parquet tables. Related Issue(s) https://github.com/delta-io/delta-rs/issues/1041, https://github.com/delta-io/delta-rs/issues/1682, https://github.com/delta-io/delta-rs/pull/1686

junjunjd avatar Nov 09 '23 09:11 junjunjd