haystack
haystack copied to clipboard
feat: FAISS in OpenSearch: Support HNSW for cosine
Related Issues
- closes #2913
Proposed Changes:
- normalize vectors manually at index time
- normalize vectors manually at query time
- use dot_product space under the hood
- make normalization more efficient by using two-dimensional numpy array instead of normalizing each vector separately
- run cosine tests for all document stores
- run all document_store integration tests (not only OpenSearch)
- fix test_faiss_and_milvus.py tests not running
pushed to another PR:
- streamline validation of embedding shape across all document stores (
BaseDocumentStore_validate_embeddings_shape
) - introduce
DenseRetriever
abstraction to facilitate usage of retrievers in document store'supdate_embeddings
How did you test it?
- made test
test_cosine_similarity
oftest_faiss_and_milvus.py
generic to run on all document stores
Notes for the reviewer
Checklist
- [x] I have read the contributors guidelines and the code of conduct
- [ ] I have updated the related issue with new insights and changes
- [x] I added tests that demonstrate the correct behavior of the change
- [x] I've used the conventional commit convention for my PR title
- [x] I documented my code
- [x] I ran pre-commit hooks and fixed any issue
Check out this pull request on
See visual diffs & provide feedback on Jupyter Notebooks.
Powered by ReviewNB