Output tokens logits

[SimonBenhamou]

Hello,

Unless I'm mistaken, I don't see any option in the translator translate_batch or generate_tokens functions to output the logits/probabilities of the generated tokens. However, this computation must be done internally before sampling the tokens, right ?

This would be very useful in order to have more control over the generation, for example for influencing the choice of specific tokens. Is there a particular reason why this is not exposed ? Could this be easily added, or am I missing something? This could for example take the form of a return_logits argument to the mentioned functions.

Thanks, Simon

[kdcyberdude]

Any update on this?

[minhthuc2502]

In the method generate_tokens, there is an option return_log_prob. It is false by default. You can activate it and get the log_prob in the result by result.log_prob

[SimonBenhamou]

@minhthuc2502 I want the distribution over the whole vocabulary (or at least top K). With your solution I only get the log prob of the token generated by the model...

This distribution must be computed though to generate the token, that's why I guess it should be "easy" to expose it...

[SimonBenhamou]

Hello,

Is there an update on this?

Thanks, Simon

[minhthuc2502]

Hello, I will include the log prob for the whole vocabulary.

[LydiaXiaohongLi]

@minhthuc2502 , is it possible to include the logits too? or return logits instead of log prob? because it is easier to calculate log prob from logits, but not the other way round. Also some model, like metricx(https://github.com/google-research/metricx) , relies on logits output.