lm-evaluation-harness icon indicating copy to clipboard operation
lm-evaluation-harness copied to clipboard

Published 20 hours ago • verified publisher icon EleutherAI

Output constrained support

Open Mihaiii opened this issue 9 months ago • 0 comments

I'd like to build a benchmark and I'd like to set up some rules based on which I can constrain the responses the LLM provides - then eval with pre-determined code (taking into consideration the predetermined constrained rules) the response.

I'm thinking at integrations with libraries like outlines or guidance.

Mihaiii avatar Apr 27 '24 19:04 Mihaiii