lm-evaluation-harness icon indicating copy to clipboard operation
lm-evaluation-harness copied to clipboard

Published 20 hours ago • verified publisher icon EleutherAI

How can I evaluate on an output file?

Open Luobots opened this issue 11 months ago • 1 comments

When I run the lm_eval with --log_samples, I got the output file. I want to only rerun the metrics on the output file, how can I do? I searched the issues, however I could not find a way.

Luobots avatar Mar 24 '24 04:03 Luobots

Hi!

We don't currently support this but we really should. Adding it as a TODO!

haileyschoelkopf avatar Mar 25 '24 12:03 haileyschoelkopf

+1 for this functionality.

harshakokel avatar May 09 '24 22:05 harshakokel