ArcticDB
ArcticDB copied to clipboard
get_description date_range truncates nanoseconds
Describe the bug
Write a dataframe with a nanosecond precision index.
Call lib.get_description(sym).date_range
.
You receive a tuple of datetime.datetime
which is only microsecond precision - nanoseconds are truncated off.
This is because we are calling:
def _from_tz_timestamp(ts, tz):
# type: (int, Optional[str])->(datetime.datetime)
return pd.Timestamp(ts).tz_localize(tz).to_pydatetime(warn=False)
Steps/Code to Reproduce
lib
can be any ArcticDB Library
.
In [24]: sym = "test_filter_datetime_nanoseconds"
...:
...: # Dataframe has three rows and a single column containing timestamps 1 nanosecond apart
...: timestamp_1 = pd.Timestamp("2023-03-15 10:30:00")
...: timestamp_0 = timestamp_1 - pd.Timedelta(1, unit="ns")
...: timestamp_2 = timestamp_1 + pd.Timedelta(1, unit="ns")
...: df = pd.DataFrame(index=[timestamp_0, timestamp_1, timestamp_2], data={"a": [1,2,3]})
In [25]: lib.write("sym", df)
Out[25]: VersionedItem(symbol='sym', library='one', data=n/a, version=1, metadata=None, host='LMDB(path=/home/alex/tmp/tst)', timestamp=1711387582004575048)
In [26]: lib.get_description("sym").date_range
Out[26]:
(datetime.datetime(2023, 3, 15, 10, 29, 59, 999999, tzinfo=datetime.timezone.utc),
datetime.datetime(2023, 3, 15, 10, 30, tzinfo=datetime.timezone.utc))
Expected Results
Date range should not drop precision.
The difficulty here will be that changing the return type is a breaking API change.
OS, Python Version and ArcticDB Version
Backend storage used
Any
Additional Context
No response