Candle cross-encoder model support

[bm777]

@LaurentMazare How can I use candle for a cross-encoder from sentence-transformers models (msmarco models: e.g. msmarco-distilroberta-base-v3)?

Does it require differents stack of implementation or can I just use the Bert example as for the embedding example?

[Ayuei]

You might be confused here. msmarco-distilroberta-base-v3 is a bi-encoder model where the query and document embeddings are computed separately. So yes, you can use the BERT embedding example.

Cross-encoder models are just the bi-encoder stack with an additional classifier layer on top, which computes the distance/relevance of the query and document at the same time. This would be something like cross-encoder/stsb-distilroberta-base. This would not require anything extra, since it's the standard BERT model with a classification layer.

[bm777]

@Ayuei Ok, did you try to use it and it computes and generates the same score as in sentence-transformers?

[fbilhaut]

@Ayuei Hi, I'm also interested in checking how Candle could be used to implement re-ranking using cross encoders. Do you think you could provide an actual example like the ones that are already available for using bi-encoders for embeddings ?

[gilljon]

Does anyone have an example handy on how to construct a cross encoder type model in candle?