Weston Pace

Feature: Data loaders for lancedb

1

### SDK All (not just pytorch) ### Description The pytorch data loader integration for lance is quite popular and useful. However, it could use some [improvements](https://github.com/lancedb/lance/issues/3727) and it is limited...

Time travel support

1

Several databases and data lakes support time travel which allows historical versions of a table to be queried. Regrettably, syntax and capabilities do not seem to be terribly consistent across...

test: add regression test for ivf/pq search

3

The test covers 5 parameters (cache, k, nprobes, refine factor, dataset size) for a total of 64 tests. It takes ~3-4 minutes to run on my system. There are other...

perf: add fast path for simple projections

4

This should improve the performance of take operations. Closes https://github.com/lance-format/lance/issues/5069

Bitmap index should support spilling

13

Currently the bitmap index just accumulates everything into a hashmap during training. If there are many unique values this can easily lead to OOM. We should spill on occasion to...

refactor: use list encoder / scheduler directly instead of replicating behavior in map encoder / scheduler

5

This is a minor simplification of the recently added map support. It does not add any functionality, only simplifies the code slightly. The current implementation creates a `StructuralMapEncoder` which mimics...

perf: rework scheduler

2

A work in progress potential scheduler rework. I think it'll need to wait until we experiment with I/O uring to see if it helps or not. Oddly it does not...