Weston Pace
Weston Pace
The [datafusion-python](https://github.com/apache/datafusion-python) project recently added support for "foreign table providers" in https://github.com/apache/datafusion-python/pull/921. We should be able to utilize this to create a foreign table provider from lance. This would make...
Our FSST configuration is not working well for code / websites. My first guess is that there are just too many opportunities for compression and the symbol table isn't able...
We byte pack control words already so the utilities to do this should be there. This is not strictly required for 2.1 beta and could be done as part of...
We currently don't compress rep-def buffers at all when mini-block encoding. This is pretty wasteful as it requires 4-bytes per value. We should, at the very least, make sure we...
As 2.1 is wrapping up I am using this issue to list features which will not make the cut for 2.1. They are candidate features for a 2.2 release. ###...
There is at least one `todo!()` (and I think several) related to 64-bit offsets in 2.1 encodings. We should be able to easily extend the 2.0 fuzz tests to include...
We should be clear that beta file versions are a bad idea in production. In particular we should be log a warning whenever they are used that states there are...
2.1 has packed struct encoding for fixed-width values. However, we should add packed struct encoding for variable-length values. Note: this does not necessarily need to block the 2.1 beta. This...
We have some 2.0 and 2.1 tests for "jumbo values" which are single strings / binary elements that are larger than 2GiB. We should verify these tests pass on 2.1....
We need to add performance benchmarks for all of the permutations listed in https://github.com/lancedb/lance/issues/3347 We should be able to achieve 100k rows / second and (ideally 3, at minimum 1)GBps...