haystack icon indicating copy to clipboard operation
haystack copied to clipboard

FAISS in OpenSearch: Support Product Quantization

Open tstadel opened this issue 2 years ago • 0 comments

As a user I want to use FAISS's product quanitzation feature with OpenSearchDocumentStore.

According to https://opensearch.org/docs/latest/search-plugins/knn/knn-index/ FAISS plug-in has two param for pq:

  • m: number of sub vectors to split the original vector
  • code_size: the number of bits to encode a sub-vector into

We only support setting m as code_size is only relevant for IVF, static for HNSW.

  • [ ] add param product_quantization_subvectors with default value None
  • [ ] if product_quantization_subvectors is not None, set pq params with m=product_quantization_subvectors
  • [ ] add check whether existing index has requested setting

tstadel avatar Jul 28 '22 16:07 tstadel