scandinavian-embedding-benchmark icon indicating copy to clipboard operation
scandinavian-embedding-benchmark copied to clipboard

Find a solution allowing empty results

Open KennethEnevoldsen opened this issue 1 year ago • 1 comments

Some models such as the "translate and embed" models can't be used for cross-lingual tasks, ideally their scores should just be nan.

I am unsure what the best solution is. We could naturally raise an error (but then it seems like it failed). We could also return nan (but then we would have to handle it in mteb, which is also

Seems like the best solution is to raise a specific error something like ModelNotValidForThisTaskError

KennethEnevoldsen avatar Feb 15 '24 08:02 KennethEnevoldsen

@x-tabdeveloping, do you have a suggestion here?

KennethEnevoldsen avatar Feb 17 '24 12:02 KennethEnevoldsen