iceberg-python icon indicating copy to clipboard operation
iceberg-python copied to clipboard

Allow parquet writer to use row group from table properties

Open amogh-jahagirdar opened this issue 1 year ago • 0 comments

Right now if the Iceberg parquet row group size table property is set, writes will fail with unsupported. It seems like a bug since the intention of _get_parquet_writer_kwargs is to surface just the kwargs for the arrow writer, and row group size isn't one of them; but we do in fact set it later here https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/pyarrow.py#L1790 as a separate argument in the write call.

amogh-jahagirdar avatar Jun 06 '24 22:06 amogh-jahagirdar