beir icon indicating copy to clipboard operation
beir copied to clipboard

Evaluate "Offline" Models

Open arendu opened this issue 11 months ago • 0 comments

Currently, beir evaluates a model by calling its encode_queries and encoder_corpus functions. These in turn call the forward methods of the model.

This works great for pytorch-based models, but not for models from other deep-learning frameworks.

To circumvent this issue, I've introduced a "general" offline model which is a static numpy array of query and corpus representations along with a mapping of desired corpus-ids for each query-id.

Thus, any model from any framework can dump its representations for queries and corpus in a numpy format which can then be evaluated using this PR.

arendu avatar Mar 19 '24 05:03 arendu