Add logprobs hfclientvllm

[tom-doerr]

This is part of https://github.com/stanfordnlp/dspy/issues/879. I tested it and it behaves the same as the other PR.

lm = dspy.HFClientVLLM(model="NurtureAI/Meta-Llama-3-8B-Instruct-32k", port=38242, url="http://localhost", max_tokens=4)
test_text = "This is a test article."
output_normal = lm(test_text)
print("output_normal:", output_normal)

output_with_logprobs = lm(test_text, logprobs=2)
print("output_with_logprobs:", output_with_logprobs)

output_normal: [' It is a test']
output_with_logprobs: [{'text': ' It is a test', 'logprobs': {'text_offset': [0, 3, 6, 8], 'token_logprobs': [-1.7945036888122559, -0.6691504716873169, -1.303508996963501, -0.7093929052352905], 'tokens': [' It', ' is', ' a', ' test'], 'top_logprobs': [{' It': -1.7945036888122559, ' This': -1.7945036888122559}, {' is': -0.6691504716873169, ' will': -2.0441503524780273}, {' a': -1.303508996963501, ' not': -1.803508996963501}, {' test': -0.7093929052352905, ' sample': -3.20939302444458}]}}]

[okhat]

Hmm this seems fine to me but it's ad-hoc.... it would be specific to this one client

[tom-doerr]

It also works already for the OpenAI API: https://github.com/stanfordnlp/dspy/pull/999. This feature wouldn't need to be documented until more clients are supported. In my initial testing the logprob feedback was much better than binary feedback. In my case non of the dataset examples passed and BootstrapFewShot gave up. With logprob feedback it was able to consistently improve over multiple iterations.

[okhat]

does it block you if we keep this open for a week or two? i need to think more about logprobs

[tom-doerr]

No doesn't block me at all, I'm using my custom branch anyway. If you are thinking about logprobs anyway, my personal wish list would be:

• Allows for access to prompt logprobs and not just completion logprobs
• Allows access to unnormalized logprobs. This can be especially useful when using logprobs to sort something

I know that not all backends can give us all that, users would have to choose a backend that supports their usecase.