evals icon indicating copy to clipboard operation
evals copied to clipboard

Exact Match template

Open JasonGross opened this issue 1 year ago • 1 comments

Describe the feature or improvement you're requesting

The documentation in eval-templates.md describes basic/match.py as Match: any([b.startswith(a) for b in B]) "[f]or a model completion a and a reference list of correct answers B. This is a poor fit for arithmetic and other algorithmic tasks, where we want the model response to exactly match some ideal answer, i.e., any([b == a for b in B])

Additional context

No response

JasonGross avatar May 26 '23 01:05 JasonGross