lance
lance copied to clipboard
`LanceFragment.to_batches` not respecting `filter` kwarg
It looks like to_batches
isn't respecting the filter kwarg
Repro
import lance
ds = lance.dataset(path)
fragments = ds.get_fragments()
for batch in fragments[0].to_batches(
batch_size=1,
filter="split == 'test'",
columns=["image", "split"],
with_row_id=True,
batch_readahead=8,
):
break
print(batch)
>>>
pyarrow.RecordBatch
image: binary
split: string
----
image: ...
split: ["train"]