[Feature] Return most probable tokens + logprobs

[deepindeed2022]

Motivation

Feature request

It would be great if the API could return a list of most probable tokens (along with their logprobs) for each step. This could be useful for many downstream tasks that require sampling of these. The option keyword is_return_logits and is_return_log_probs in the current interface are not available, see issue

Details

Many LLM-based applications benefit not just from having the most probable token, but also a list of output probabilities.

• Some classification tasks can be done by selecting only the most probable token, but only when this is actually one of the possible labels. Having the output logits, makes it possible to mitigate this and just select the token that's actually one of the options (having logit bias available will also help with this, but that's another feature)

  lm-evaluation-harness relies on having the output logits for its multiple-choice benchmarks. Having this feature will allow using this server to speed up benchmarking tasks.

• In many classification use-cases you want the probability of the answer. Without logit_bias, you could use top_k, hope your prompting is sufficient such that the first k tokens are all your other classes and that the softmax is only taken over these tokens. It is, however, impossible to know for sure.

• For the same reason as the previous point, binary ranking is unreliable. Multi-class ranking is nigh impossible (except for using high best-of and temperature to hope you get all label's probabilities).

Related resources

• https://github.com/InternLM/lmdeploy/issues/323
• https://github.com/huggingface/text-generation-inference/pull/617

Additional context

No response

[deepindeed2022]

Hi, any ideas or plans? @lvhan028 @lzhangzz

[dawnranger]

Same request.

@lvhan028 any plans about this feature?

[lvhan028]

This feature has been supported since v0.4.0 May check this PR for some examples

[deepindeed2022]

Confirm on v0.4.0, the output with logprobs when request with logprobs values.

RequestFuncOutput(text='To find the total number of books needed, multiply the number of rows by the number of columns: 6 rows \* 6 columns = <<6\\*6=36>>36 books\nTo find the number of books Wendy needs to add, subtract the current number of books from the total number of books: 36 books - 20 books = <<36-20=16>>16 books\nWendy will need to carry 16 books to complete her task.', logprobs={'text_offset': [0, 2, 7, 11, 17, 24, 27, 33, 40, 41, 50, 54, 61, 64, 69, 72, 76, 83, 86, 94, 95, 96, 97, 102, 104, 105, 106, 107, 115, 117, 120, 121, 122, 123, 124, 125, 126, 127, 129, 130, 131, 137, 138, 140, 145, 149, 156, 159, 165, 170, 171, 177, 180, 184, 185, 194, 198, 206, 213, 216, 222, 227, 231, 237, 244, 247, 253, 254, 255, 256, 257, 263, 265, 266, 267, 268, 274, 276, 279, 280, 281, 282, 283, 284, 285, 286, 287, 289, 290, 291, 297, 298, 299, 302, 303, 308, 313, 316, 322, 323, 324, 325, 331, 334, 343, 347, 352], 'token_logprobs': [0.0, -0.1797899305820465, 0.0, -0.15967382490634918, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.12331568449735641, 0.0, 0.0, 0.0, 0.0, 0.0, -0.13043580949306488, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.43511733412742615, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.4207225739955902, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.9808534383773804, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -1.3610799312591553, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'tokens': ['To', ' find', ' the', ' total', ' number', ' of', ' books', ' needed', ',', ' multiply', ' the', ' number', ' of', ' rows', ' by', ' the', ' number', ' of', ' columns', ':', ' ', '6', ' rows', ' \', '', ' ', '6', ' columns', ' =', ' <<', '6', '\', '', '6', '=', '3', '6', '>>', '3', '6', ' books', '\n', 'To', ' find', ' the', ' number', ' of', ' books', ' Wend', 'y', ' needs', ' to', ' add', ',', ' subtract', ' the', ' current', ' number', ' of', ' books', ' from', ' the', ' total', ' number', ' of', ' books', ':', ' ', '3', '6', ' books', ' -', ' ', '2', '0', ' books', ' =', ' <<', '3', '6', '-', '2', '0', '=', '1', '6', '>>', '1', '6', ' books', '\n', 'W', 'end', 'y', ' will', ' need', ' to', ' carry', ' ', '1', '6', ' books', ' to', ' complete', ' her', ' task', '.'], 'top_logprobs': [{'To': 0.0}, {' find': -0.1797899305820465, ' place': -1.8045152425765991}, {' the': 0.0}, {' total': -0.15967382490634918, ' number': -1.913396954536438}, {' number': 0.0}, {' of': 0.0}, {' books': 0.0}, {' needed': 0.0}, {',': 0.0}, {' multiply': 0.0}, {' the': 0.0}, {' number': 0.0}, {' of': 0.0}, {' rows': 0.0}, {' by': 0.0}, {' the': 0.0}, {' number': 0.0}, {' of': 0.0}, {' columns': 0.0}, {':': 0.0}, {' ': 0.0}, {'6': 0.0}, {' rows': 0.0}, {' \': 0.0}, {'': 0.0}, {' ': 0.0}, {'6': 0.0}, {' columns': 0.0}, {' =': 0.0}, {' <<': 0.0}, {'6': 0.0}, {'\': 0.0}, {'': 0.0}, {'6': 0.0}, {'=': 0.0}, {'3': 0.0}, {'6': 0.0}, {'>>': 0.0}, {'3': 0.0}, {'6': 0.0}, {' books': 0.0}, {'\n': -0.12331568449735641, '.': -2.15403151512146}, {'To': 0.0}, {' find': 0.0}, {' the': 0.0}, {' number': 0.0}, {' of': 0.0}, {' books': -0.13043580949306488, ' additional': -2.1013829708099365}, {' Wend': 0.0}, {'y': 0.0}, {' needs': 0.0}, {' to': 0.0}, {' add': 0.0}, {',': 0.0}, {' subtract': -0.43511733412742615, ' multiply': -1.0418219566345215}, {' the': 0.0}, {' current': 0.0}, {' number': 0.0}, {' of': 0.0}, {' books': 0.0}, {' from': 0.0}, {' the': -0.4207225739955902, ' this': -1.0687785148620605}, {' total': 0.0}, {' number': 0.0}, {' of': 0.0}, {' books': 0.0}, {':': 0.0}, {' ': 0.0}, {'3': 0.0}, {'6': 0.0}, {' -': -0.46998918056488037, ' books': -0.9808534383773804}, {' -': 0.0}, {' ': 0.0}, {'2': 0.0}, {'0': 0.0}, {' books': 0.0}, {' =': 0.0}, {' <<': 0.0}, {'3': 0.0}, {'6': 0.0}, {'-': 0.0}, {'2': 0.0}, {'0': 0.0}, {'=': 0.0}, {'1': 0.0}, {'6': 0.0}, {'>>': 0.0}, {'1': 0.0}, {'6': 0.0}, {' books': 0.0}, {'': -0.29623013734817505, '\n': -1.3610799312591553}, {'W': 0.0}, {'end': 0.0}, {'y': 0.0}, {' will': 0.0}, {' need': 0.0}, {' to': 0.0}, {' carry': 0.0}, {' ': 0.0}, {'1': 0.0}, {'6': 0.0}, {' books': 0.0}, {' to': 0.0}, {' complete': 0.0}, {' her': 0.0}, {' task': 0.0}, {'.': 0.0}]}, finish_reason='stop', success=True, prompt_len=1143, output_len=107, error='')