elastik-nearest-neighbors
elastik-nearest-neighbors copied to clipboard
alexklibisz
Filter documents for searching
One of the biggest drawbacks of libs like annoy or Faiss is that it doesn't allow filtering of documents to be searched before finding nearest neighbours. This is something that can be solved using the changes I have suggested
@jainaayush05 I don't think the cosineDistance part will work. I'm pretty sure that the LSH hashing implementation would have to change in order for cosine distance to make sense. Do you have results to suggest otherwise?
@jainaayush05 Any thoughts on the cosine distance not working with this implementation of LSH? @alexklibisz If the filtering of documents part of this pull request work, can that be added to your library?
The implementation of lsh is different for cosine and euclidean similarity. If you are interested, this video series is a fantastic deep dive: https://youtu.be/tlGocuyQ7F0
It's certainly possible to implement but it's not plug and play with the current implementation. Also, the implementation for euclidean in this plugin is actually more complicated than it needs to be.
I'm not maintaining any of this code anymore, so I'd rather not merge any PRs. I don't really consider it a library so much as it's a proof of concept. I think there are one or two forks floating around that other people have extended and might be maintaining. Would love to maintain it but there's only so much time in a day, and my current daily work is unrelated.
On Wed, Aug 28, 2019, 02:47 Sai [email protected] wrote:
@jainaayush05 https://github.com/jainaayush05 Any thoughts on the cosine distance not working with this implementation of LSH? @alexklibisz https://github.com/alexklibisz If the filtering of documents part of this pull request work, can that be added to your library?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/alexklibisz/elastik-nearest-neighbors/pull/10?email_source=notifications&email_token=AB5E27GP5FNZGVSQNAIWZF3QGYNO5A5CNFSM4GJHRQD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5KCFVI#issuecomment-525607637, or mute the thread https://github.com/notifications/unsubscribe-auth/AB5E27BQLJ3RVOG7CIBQ363QGYNO5ANCNFSM4GJHRQDQ .