ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

get_description date_range truncates nanoseconds

Open poodlewars opened this issue 11 months ago • 0 comments

Describe the bug

Write a dataframe with a nanosecond precision index.

Call lib.get_description(sym).date_range.

You receive a tuple of datetime.datetime which is only microsecond precision - nanoseconds are truncated off.

This is because we are calling:

def _from_tz_timestamp(ts, tz):
    # type: (int, Optional[str])->(datetime.datetime)
    return pd.Timestamp(ts).tz_localize(tz).to_pydatetime(warn=False)

Steps/Code to Reproduce

lib can be any ArcticDB Library.

In [24]:     sym = "test_filter_datetime_nanoseconds"
    ...: 
    ...:     # Dataframe has three rows and a single column containing timestamps 1 nanosecond apart
    ...:     timestamp_1 = pd.Timestamp("2023-03-15 10:30:00")
    ...:     timestamp_0 = timestamp_1 - pd.Timedelta(1, unit="ns")
    ...:     timestamp_2 = timestamp_1 + pd.Timedelta(1, unit="ns")
    ...:     df = pd.DataFrame(index=[timestamp_0, timestamp_1, timestamp_2], data={"a": [1,2,3]})

In [25]: lib.write("sym", df)
Out[25]: VersionedItem(symbol='sym', library='one', data=n/a, version=1, metadata=None, host='LMDB(path=/home/alex/tmp/tst)', timestamp=1711387582004575048)

In [26]: lib.get_description("sym").date_range
Out[26]: 
(datetime.datetime(2023, 3, 15, 10, 29, 59, 999999, tzinfo=datetime.timezone.utc),
 datetime.datetime(2023, 3, 15, 10, 30, tzinfo=datetime.timezone.utc))

Expected Results

Date range should not drop precision.

The difficulty here will be that changing the return type is a breaking API change.

OS, Python Version and ArcticDB Version

Backend storage used

Any

Additional Context

No response

poodlewars avatar Mar 25 '24 17:03 poodlewars