iceberg-python

iceberg-python copied to clipboard

Bump to PyArrow 17.0.0

12

Fokko

[feat] Unify implementation of `to_arrow` and `to_arrow_batch_reader`

2

### Feature Request / Improvement Currently, these two functions are performing similar tasks; the only difference is the output format. Let's unify the implementations so they don't diverge. For example,...

kevinjqliu

[bug] `to_arrow_batch_reader` does not respect the given limit, returning more records than specified

3

### Apache Iceberg version main (development) ### Please describe the bug 🐞 ### `to_arrow_batch_reader` bug The bug is in [project_batches](https://github.com/apache/iceberg-python/blob/d8b5c17cadbc99e53d08ade6109283ee73f0d83e/pyiceberg/io/pyarrow.py#L1457-L1475), specifically with the way yield interacts with the two for-loops....

kevinjqliu

[Bug] Cannot use PyIceberg with multiple FS

15

### Apache Iceberg version main (development) ### Please describe the bug 🐞 PyIceberg assumes the same FS implementation is used for reading both metadata and data. However, I want to...

kevinjqliu

This resolves #998 , where duplicate files are added with add_files method, handles 2 cases: 1. Files list is not unique 2. One of the files added is already referenced...

amitgilad3

Fields with mixed datatypes

1

### Question How would I go about using a field with mixed datatypes? Is that recommended/possible? I am a fan of tall-tidy data and am wondering how to properly go...

jayceslesar

HA HMS support

3

### Changes Proposed in this PR Support a HA HMS URI such as `uri: thrift://hms-1:9083,thrift://hms-2:9083` currently is not supported. This change supports HA HMS were each entry will be tried...

awdavidson

Support execution in Windows using Local File System and NFS

4

When running PyIceberg in Windows, `pyiceberg.io.pyarrow.PyArrowFileIO.parse_location()` incorrectly resolve Windows path, e.g. `C:\foobar` as a scheme of `C` and causes `Unrecognized filesystem type in URI` error later. If using `file://` scheme,...

rfung777

### Feature Request / Improvement As of now, `limit` is checked only after an entire parquet file is read. https://github.com/apache/iceberg-python/blob/d8b5c17cadbc99e53d08ade6109283ee73f0d83e/pyiceberg/io/pyarrow.py#L1360-L1390 Optimization to pushdown limit to the parquet reading level For...

kevinjqliu

Prevent `add_files` from adding a file that's already referenced by the Iceberg Table

3

### Feature Request / Improvement Currently `add_files` doesn't have a check to prevent adding an object that's already referenced by the Iceberg Table. We should include these two checks to...

sungwy