lance icon indicating copy to clipboard operation
lance copied to clipboard

`LanceFragment.to_batches` not respecting `filter` kwarg

Open tonyf opened this issue 6 months ago • 1 comments

It looks like to_batches isn't respecting the filter kwarg

Repro

import lance

ds = lance.dataset(path)
fragments = ds.get_fragments()

for batch in fragments[0].to_batches(
    batch_size=1, 
    filter="split == 'test'", 
    columns=["image", "split"], 
    with_row_id=True, 
    batch_readahead=8,
):
    break

print(batch)
>>>
pyarrow.RecordBatch
image: binary
split: string
----
image: ...
split: ["train"]

tonyf avatar Aug 22 '24 23:08 tonyf