lm-evaluation-harness icon indicating copy to clipboard operation
lm-evaluation-harness copied to clipboard

feat: COT trace response handling in evaluator and model classes

Open hhh2210 opened this issue 4 months ago • 1 comments

  • Added support for storing raw generations in HFLM and VLLM models.
  • Updated the evaluator to log warnings when the length of raw generations does not match processed responses.
  • Modified response collection to include raw responses when available.

This improves the evaluation process by allowing access to the original generated outputs alongside processed responses.

hhh2210 avatar Aug 03 '25 04:08 hhh2210

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Aug 03 '25 04:08 CLAassistant