dspy
dspy copied to clipboard
AI feedback using logprobs
Whenever I use a LM to evaluate something I ask a Yes/No question and do logprob_yes - logprob_no
. This worked very well for me in the past. How can I access logprobs in my metric function?
Hi @tom-doerr , logprobs are indeed quite useful. Currently, DSPy doesn't support returning additional parameters besides the response content, but you can extract the logprobs from the response (subject to different LM configurations).
Feel free to push a PR that enables some flag for returning completions with logprobs while maintaining internal behavior!