lm-evaluation-harness icon indicating copy to clipboard operation
lm-evaluation-harness copied to clipboard

Published 20 hours ago • verified publisher icon EleutherAI

feat: Numeric bench

Open Gresham429 opened this issue 8 months ago • 4 comments

Gresham429 avatar Mar 24 '25 04:03 Gresham429

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Mar 24 '25 04:03 CLAassistant

Hi! Thanks for the PR. Looks good to merge once you run the pre-commit:

pip install pre-commit
pre-commit install
pre-commit run --all-files

baberabb avatar Mar 25 '25 03:03 baberabb

Hi! Thanks for the PR. Looks good to merge once you run the pre-commit:

pip install pre-commit
pre-commit install
pre-commit run --all-files

thanks for helping, I've been a bit busy lately, but I'll fix these bugs in a few days.

Gresham429 avatar Mar 25 '25 05:03 Gresham429

thanks for helping, I've been a bit busy lately, but I'll fix these bugs in a few days.

no problem! Also noticed you have a system prompt, but using it in doc_to_text. Wasn't sure if you're aware, you can use the description field for it. It will format to the system prompt in case the chat template is used, otherwise prepended to the doc_to_text (before the few-shots, if used). see parameters

baberabb avatar Mar 25 '25 06:03 baberabb