how to insert rows and datafile to the front of the dataset in lance?

[zeddit]

when managing time series data, there will be the cases when we want to insert some rows to the front of the dataset.

for example, I have a dataset which with a column named date and is ranging from 2020-01-01 to 2020-12-31. then I want to insert some rows ranging from 2019-01-01 to 2019-12-31 into the dataset.

because I want to query the data in a correct time order, which means the historical data should be placed in the front rows.

however, I found the api of lance write_dataset support only append mode, which could only add data to the tail of the table.

so how could I achieve the correct time order in the underlying table format.

great thanks!

[zeddit]

besides, update method will also mess up the order of rows for table or dataset we need order. update will insert the new row to the end of the table.

how could I maintain the order with lance or lancedb. it would be performance issues if I pull all of the data and reorder the whole rows.