Weston Pace

feat: standalone vector transform stage

2

This makes the vector transform stage a standalone operation.

Bitmap indices are a type of scalar index that works well when the cardinality of a column is low. A bitmap index stores a compressed bitmap (0 = no match...

perf: use v2 to write the sorted IVF partition files

5

The old v1 approach created a row group for every partition (including empty partitions). This approach converts the input into a `List` array where each row is a partition of...

Or we should always save them to the tmp folder and clearly advertise that they are saved. Otherwise it ends up being somewhat awkward that we have an option to...

PQ assignment stage `HashMap` lookups are too slow at 1B rows

1

We don't need a HashMap anyways. Since we're mapping from row address to partition id we can use a `Vec` where each lookup is `map[fragment_id][row_offset]`. From some experimentation this is...

Shuffling stage is too slow

2

When training an index the shuffling stage runs too slowly. This seems to be because we are storing 1 batch per partition (even empty partitions need a batch) and there...

Segmentation fault training index on high-cpu system

5

When training in index on a system with 250+ CPUs a segmentation fault is generated during partition assignment. The backtrace of the offending thread was incredibly deep (5000+ calls) and...

While working on https://github.com/lancedb/lance/pull/2492 I, at one point, horribly broke the index (was writing out jumbled row ids) but I could also get all the tests to pass. We should...

Previously, we used batch schema to check if incoming data was compatible with the dataset. This has a few problems: * You couldn't add a non-nullable batch to a nullable...