Daft icon indicating copy to clipboard operation
Daft copied to clipboard

Reading only the partition column from an Iceberg/Delta table fails

Open jaychia opened this issue 1 year ago • 0 comments

Describe the bug

When table formats such as Iceberg and Delta Lake store the data for a partition column, they will strip the column from the actual Parquet data files that they write out.

NOTE: Seems like Iceberg only does this for identity transformed partition columns

Daft is then unable to read this data when performing a query such as: daft.read_delta_lake(...).select("partition_column")

jaychia avatar Apr 15 '24 17:04 jaychia