datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Allow for writing hive-style partitions with datatypes beyond just Utf8 and Dictionary

Open Omega359 opened this issue 1 year ago • 0 comments

Is your feature request related to a problem or challenge?

Currently in demux::compute_partition_keys_by_row the only supported types for writing out partitions seems to be just DataType::Utf8 and DataType::Dictionary(_, _). I think there is opportunity to support a number of other DataTypes such as int/uint 8/32/64 types, Date32 (with a fixed format 'yyyy-MM-dd') and bool.

Describe the solution you'd like

Code and tests for writing out hive-style partitions includes additional datatypes beyond just utf8 and Dictionary

Describe alternatives you've considered

Cast field to utf8 prior to output.

Additional context

No response

Omega359 avatar Aug 28 '24 17:08 Omega359