scandinavian-embedding-benchmark icon indicating copy to clipboard operation
scandinavian-embedding-benchmark copied to clipboard

Bootstrapping scores for an uncertainty estimate

Open KennethEnevoldsen opened this issue 1 year ago • 1 comments

The current implementation of the evaluators only gives a singular score. This makes it hard to see the uncertainty in the scores.

A potential solution is bootstrapping on the document level and recalculating the scores.

KennethEnevoldsen avatar Jan 25 '24 07:01 KennethEnevoldsen

We have started including this within MTEB. So it is likely that future versions of SEB will include it as well.

KennethEnevoldsen avatar Jul 18 '24 12:07 KennethEnevoldsen