added CrossEncoderReranker with HuggingFaceCrossEncoder and SagemakerEndpointCrossEncoder

[kennethchoe]

• Description: Support reranking based on cross encoder models available from HuggingFace.
  • Added CrossEncoder schema
  • Implemented HuggingFaceCrossEncoder and SagemakerEndpointCrossEncoder
  • Implemented CrossEncoderReranker that performs similar functionality to CohereRerank
  • Added cross-encoder-reranker.ipynb to demonstrate how to use it. Please let me know if anything else needs to be done to make it visible on the table-of-contents navigation bar on the left, or on the card list on retrievers documentation page.
• Issue: N/A
• Dependencies: None other than the existing ones.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	🛑 Canceled (Inspect)			Mar 31, 2024 8:51pm

[kennethchoe]

The issue pointed out by automated check Error: docs/docs/integrations/retrievers/cross-encoder-reranker.ipynb:1:1: I001 Import block is un-sorted or un-formatted is due to this code block:

#!pip install faiss sentence_transformers

# OR  (depending on Python version)

#!pip install faiss-cpu sentence_transformers

which is intentional. It can be also found from Cohere Reranker documentation.

[austinmw]

Interested to see this feature added!

[kennethchoe]

The added documentation cross-encoder-reranker.ipynb does not include detailed instruction on how to deploy cross encoder model to SageMaker. I think it is out of scope, so I made a separate post on https://huggingface.co/blog/kchoe/deploy-any-huggingface-model-to-sagemaker

[kennethchoe]

Pinging the moderators - @hwchase17 @baskaryan - for updates on the review of this pull request. Thank you!

[shell-nlp]

What can support cross encoders?

[kennethchoe]

What can support cross encoders?

https://huggingface.co/cross-encoder

But there are more models that support cross encoder. The one I used https://huggingface.co/BAAI/bge-reranker-base does not show up in that. So also try https://huggingface.co/models?sort=trending&search=cross-en

[shell-nlp]

May I ask when will this PR be merged? 请问什么时候合并这个PR？

[shell-nlp]

What can support cross encoders?什么可以支持交叉编码器？

https://huggingface.co/cross-encoder

But there are more models that support cross encoder. The one I used https://huggingface.co/BAAI/bge-reranker-base does not show up in that.但还有更多型号支持交叉编码器。我使用的 https://huggingface.co/BAAI/bge-reranker-base 没有出现在其中。 So also try https://huggingface.co/models?sort=trending&search=cross-en所以也尝试 https://huggingface.co/models?sort=trending&search=cross-en

This branch has conflicts that must be resolved 这个分支有必须解决的冲突

[kennethchoe]

Conflict is resolved. Pinging the moderators again for your thoughts - @hwchase17 @baskaryan.

[kennethchoe]

Pinging the moderators as a reminder - @hwchase17 @baskaryan