verifiers
verifiers copied to clipboard
Ability to skip rollout and load cached responses
Would be super useful for debugging a rubric and unit testing if we one could bypass the rollout function and simply load past responses, probably from results.jsonl file in outputs.
+1 on that. esp with different env args. In my case i have an env where the parser behaves differently to support thinking models, and i forgot one too many times to enable the think flag :'(
i.e.
# on load_environment
if use_think:
parser = vf.ThinkParser()
else:
parser = vf.Parser()
would be happy to open a PR if it makes sense.