dspy icon indicating copy to clipboard operation
dspy copied to clipboard

Published 20 hours ago • verified publisher icon stanfordnlp

AI feedback using logprobs

Open tom-doerr opened this issue 1 year ago • 3 comments

Whenever I use a LM to evaluate something I ask a Yes/No question and do logprob_yes - logprob_no. This worked very well for me in the past. How can I access logprobs in my metric function?

tom-doerr avatar Apr 20 '24 22:04 tom-doerr

Hi @tom-doerr , logprobs are indeed quite useful. Currently, DSPy doesn't support returning additional parameters besides the response content, but you can extract the logprobs from the response (subject to different LM configurations).

Feel free to push a PR that enables some flag for returning completions with logprobs while maintaining internal behavior!

arnavsinghvi11 avatar Apr 27 '24 23:04 arnavsinghvi11