lance
lance copied to clipboard
lance could write but could not filter on data with type like `TIME` and `DURATION`
I have tested that lance could able to save and load data type like TIME and DURATION correctly.
import pandas as pd
import lance
import pyarrow as pa
table = pa.table({
'date': pa.array([pd.to_datetime('2020-01-01').date()]),
'time': pa.array([pd.to_datetime('2020-01-01T19:03:01').time()]),
'datetime': pa.array([pd.to_datetime('2020-01-01T19:03:01')])
})
print(table)
dataset = lance.write_dataset(table, "timetest")
print(lance.dataset("timetest").to_table().to_pandas())
====
pyarrow.Table
date: date32[day]
time: time64[us]
datetime: timestamp[us]
----
date: [[2020-01-01]]
time: [[19:03:01.000000]]
datetime: [[2020-01-01 19:03:01.000000]]
and timedelta in pandas or duration in arrow.
Data columns (total 56 columns):
# Column Dtype
--- ------ -----
0 date datetime64[ns]
1 time timedelta64[ns]
====
pyarrow.Table
date: timestamp[ns]
time: duration[ns]
an example shows below
However, the current filter query only support using Literals and the literals only support DATE and TIMESTAMP data type, which makes query on TIME and DURATION type hard.