databrickslabs
[Feature] Logprobs
Hi, thanks for sharing the code and models! May I ask what's an easy way to obtain the logprobs of each chosen tokens (not necessarily the entire vocab) during generation? Thanks!
I haven't tried this, but I think you can just use compute_transition_scores for this in the transformers API, like at https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationMixin.compute_transition_scores.example
Just to report back, compute_transition_scores is exactly what I needed. Thanks for the suggestion.
One thing worth mentioning is that, when locating logprobs per token (rather than strings that we can strip), one needs to change this to response_pos + 2 because there is an extra \n after the response token (as defined here).
Just to report back,
compute_transition_scoresis exactly what I needed. Thanks for the suggestion. One thing worth mentioning is that, when locating logprobs per token (rather than strings that we can strip), one needs to change this toresponse_pos + 2because there is an extra\nafter the response token (as defined here).
@xingdi-eric-yuan, would you help share how you achieve the change to return the scores for each token?
