evals icon indicating copy to clipboard operation
evals copied to clipboard

Add run eval workflow

Open mscheong01 opened this issue 1 year ago • 0 comments

Hi, I have added a workflow to this repository based on test_eval.yaml. This workflow enables the detection of PR comments including /run {OPENAI_MODEL_NAME} and runs the new evals included in the PR on the given model. It is designed to run only for users with write/admin permissions.

The purpose of this workflow is to allow for public running of evaluations on GPT-4 and sharing the results with evaluation writers. Additionally, we could potentially expand this workflow to report the run results to the PR.

While I am unsure if this repository is currently accepting contributions other than evals, I believe this workflow could be a valuable addition. If there are any concerns, please do let me know and I will close this PR accordingly.

mscheong01 avatar Mar 23 '23 14:03 mscheong01