arrow-julia icon indicating copy to clipboard operation
arrow-julia copied to clipboard

Published 20 hours ago • verified publisher icon apache

Store ZDT with a UTC, not local, timestamp

Open tpgillam opened this issue 2 years ago • 2 comments

Closes #327

This changes the representation of zone-aware datetimes to store a UTC timestamp + zone. Previously Arrow.jl used the local timestamp + zone, which leads to ambiguity failures around DST changes. More importantly, it seems like this change is required to be consistent with the standard (see discussion here)

If this is indeed the appropriate fix, I'd appreciate thoughts on how we can make this change safely / with minimal disruption!

  • This is clearly a breaking change, in that any new version of Arrow.jl will be incompatible with arrow files (containing zone-aware datetimes) written with older versions of Arrow.jl.

  • Other than changing the package version in the usual way, is there any other kind of version info that's available? I'm wondering if it's possible prevent users from loading a file written with an older / incompatible version of Arrow.jl

  • I noticed a pyarrow_rountrip.jl file in the tests. I was hoping that this might provide some mechanism to test for interop wrt. zoned datetimes. However, I'm actually not sure that this is its purpose (it doesn't seem to be run as part of CI, or indeed have any @tests. Can someone (@quinnj ? ) indicate if this kind of test is possible right now please? Thanks!

tpgillam avatar Aug 08 '22 08:08 tpgillam

I've made a PR in https://github.com/tpgillam/arrow-julia/pull/1 to try to make this change backwards compatible.

ericphanson avatar Aug 10 '22 12:08 ericphanson

I'm also wondering about how (if at all) the native Timestamp logic interacts with any of this. For example here:

https://github.com/apache/arrow-julia/blob/52bfe1f72fa72b0b2d150efeeeb9edacd853f96f/src/metadata/Schema.jl#L319-L336

I think (based on #303 ) that native-arrow-time-with-timezone should somewhere get converted to a ZonedDateTime on reading, but can't figure out if that's a different code path, or is also affected by the issue.

tpgillam avatar Aug 10 '22 13:08 tpgillam

Just bumping this - I'm running into the same issue

snakch avatar Nov 18 '22 18:11 snakch

this looks fine to me -- @tpgillam could you merge in https://github.com/tpgillam/arrow-julia/pull/1 and convert this PR from draft?

nickrobinson251 avatar Nov 18 '22 18:11 nickrobinson251

@nickrobinson251 I have rebased this PR on the latest main - so should be good for a test run & hopefully merging now.

tpgillam avatar Dec 03 '22 10:12 tpgillam