evals
evals copied to clipboard
openai
Update documentation regarding custom evals
docs/custom-eval.md discusses how to make an eval with custom code, however it is not mentioned anywhere in docs/custom-eval.md or docs/build-eval.md that these evals are not being accepted currently. It took me a bit of digging around to figure out that these PR's are not being accepted currently. I realized this first seeing that the folder mentioned in the docs (/evals/evlsuite) is devoid of any contributions from others. Then I dug around and found out that #520 was denied because of this reason.
I'd love it if this restriction was lifted because I have an idea for an extremely useful use-case (summarizing text to a specific range of word count e.g. 50-60 words) that does require custom code evals.
Do we know if this is still the case? Just spent the last few days writing a custom eval and found this while going to submit the PR.
Custom-eval.md is obsolete after the refactor, right? Eg, the evals.check_sampled_text used in arithmetic.py has been removed from Eval.
See here - https://github.com/openai/evals/pull/579/files#diff-b14c6017190edd8585b6766159928c2a27512bab71a3ce84a027fc283dcfb240L101
I think it's now using the OpenAI completion work.
Custom-eval.md is obsolete after the refactor, right? Eg, the evals.check_sampled_text used in arithmetic.py has been removed from Eval.
Yes, that does make sense! I was trying to find a few of those functions in api.py but couldn't. I guess the docs need to be updated.