delta-rs icon indicating copy to clipboard operation
delta-rs copied to clipboard

Support parquet files with nanoseconds timestamps when converting a Parquet table to a Delta table

Open junjunjd opened this issue 8 months ago • 3 comments

Description

When converting from ArrowSchema to deltalake::Schema, the try_from method will return an error if the arrow schema has a Timestamp(nanosecond). This causes the convert_to_delta function to fail. It makes sense to support parquet files with nanoseconds timestamps when converting a Parquet table to a Delta table.

Use Case

Convert parquet files in rust/tests/data/delta-1.2.1-only-struct-stats to a Delta table.

Related Issue(s) https://github.com/delta-io/delta-rs/issues/1041, https://github.com/delta-io/delta-rs/issues/1682, https://github.com/delta-io/delta-rs/pull/1686

junjunjd avatar Oct 13 '23 02:10 junjunjd