iceberg-python
iceberg-python copied to clipboard
fix: allow reading pyarrow timestamptz as iceberg timestamp
Rationale for this change
Closes #2663 Relates to #2333 which allow reading pyarrow timestamp as iceberg timestamptz
This PR allows pyarrow timestamptz to be read as iceberg timestamp.
Although this configuration does not conform to the Iceberg spec, the change aligns PyIceberg's behavior with Spark when reading mismatched types (timestamptz as iceberg timestamp and timestamp as iceberg timestamptz)
Are these changes tested?
Yes
Are there any user-facing changes?
No
I'm comfortable with having this patch in the read path, but it looks like we're also implicitly adding this to the write path:
https://github.com/apache/iceberg-python/blob/e759044601feaf34c99b5c154322b8b52e4f6a30/pyiceberg/io/pyarrow.py#L2576
I'm reluctant to add it there since we want to enforce the difference between timestamp and timestamptz and not silently ignore it.
thanks! restricting the write side makes sense. Let me take a look at how we can do that