haystack
haystack copied to clipboard
FAISS in OpenSearch: Support Product Quantization
As a user I want to use FAISS's product quanitzation feature with OpenSearchDocumentStore
.
According to https://opensearch.org/docs/latest/search-plugins/knn/knn-index/ FAISS plug-in has two param for pq
:
-
m
: number of sub vectors to split the original vector -
code_size
: the number of bits to encode a sub-vector into
We only support setting m
as code_size
is only relevant for IVF, static for HNSW.
- [ ] add param
product_quantization_subvectors
with default valueNone
- [ ] if
product_quantization_subvectors
is notNone
, set pq params with m=product_quantization_subvectors - [ ] add check whether existing index has requested setting