lance icon indicating copy to clipboard operation
lance copied to clipboard

lance could write but could not filter on data with type like `TIME` and `DURATION`

Open zeddit opened this issue 1 year ago • 0 comments

I have tested that lance could able to save and load data type like TIME and DURATION correctly.

import pandas as pd
import lance
import pyarrow as pa

table = pa.table({
    'date': pa.array([pd.to_datetime('2020-01-01').date()]),
    'time': pa.array([pd.to_datetime('2020-01-01T19:03:01').time()]),
    'datetime': pa.array([pd.to_datetime('2020-01-01T19:03:01')])
})
print(table)
dataset = lance.write_dataset(table, "timetest") 
print(lance.dataset("timetest").to_table().to_pandas())
====
pyarrow.Table
date: date32[day]
time: time64[us]
datetime: timestamp[us]
----
date: [[2020-01-01]]
time: [[19:03:01.000000]]
datetime: [[2020-01-01 19:03:01.000000]]

and timedelta in pandas or duration in arrow.

Data columns (total 56 columns):
 #   Column      Dtype          
---  ------      -----          
 0   date        datetime64[ns] 
 1   time        timedelta64[ns]
====
pyarrow.Table
date: timestamp[ns]
time: duration[ns]

an example shows below 截屏2024-04-10 13 53 05

However, the current filter query only support using Literals and the literals only support DATE and TIMESTAMP data type, which makes query on TIME and DURATION type hard.

zeddit avatar Apr 10 '24 05:04 zeddit