iceberg-python icon indicating copy to clipboard operation
iceberg-python copied to clipboard

fix: allow reading pyarrow timestamptz as iceberg timestamp

Open kevinjqliu opened this issue 1 month ago • 2 comments

Rationale for this change

Closes #2663 Relates to #2333 which allow reading pyarrow timestamp as iceberg timestamptz

This PR allows pyarrow timestamptz to be read as iceberg timestamp.

Although this configuration does not conform to the Iceberg spec, the change aligns PyIceberg's behavior with Spark when reading mismatched types (timestamptz as iceberg timestamp and timestamp as iceberg timestamptz)

Are these changes tested?

Yes

Are there any user-facing changes?

No

kevinjqliu avatar Nov 09 '25 16:11 kevinjqliu

I'm comfortable with having this patch in the read path, but it looks like we're also implicitly adding this to the write path:

https://github.com/apache/iceberg-python/blob/e759044601feaf34c99b5c154322b8b52e4f6a30/pyiceberg/io/pyarrow.py#L2576

I'm reluctant to add it there since we want to enforce the difference between timestamp and timestamptz and not silently ignore it.

Fokko avatar Nov 10 '25 17:11 Fokko

thanks! restricting the write side makes sense. Let me take a look at how we can do that

kevinjqliu avatar Nov 11 '25 20:11 kevinjqliu