mteb
mteb copied to clipboard
Allow Evaluation should return results from cached runs instead of skipping them
The evaluation do not run tasks twice, that is smart as it saves ressources but why is the task deleted ??
I am using the python SDK and spent several minutes wondering why my code didn't work anymore (do not return any result) and that was the culprit.
I'd thought that the package should use already cached results instead of just not returning anything.
Any design justification ? Is it a bug ?