IVF_SQ index

[fferroni]

Hello, Do you support or plan on supporting SQ compression? I see you support PQ compression. FAISS also supports milder SQ compression which for my use-cases works better. Wondering if this is on the road map. If not, where could I look to adding it?

[eddyxu]

Hey, @fferroni

Thanks for looking into lance. We would love to and open to supporting more Faiss index types in Lance. Our current rodmap is improving IVF_PQ indexing perf and caching behavior, as well as implementing a performant DiskANN, as near term roadmap. IVF_SQ or IVF_PQfs would be nice to have, we just don't have a good estimation when we can get on to it yet.

If you are interested, you can take a look at https://github.com/lancedb/lance/tree/main/rust/src/index/vector directory, with our existing index implementations. Please take a look, we will be super happy to accept contributions from the community. Even giving us some rough guidance and estimations would help us to plan on the feature too.

Feel free to leave feedback here or shot us an email at [email protected].