New feature request: Topk log probabilities with TokenIDs for each position of input sequence.

[salaki]

trafficstars

System Info

H100

Who can help?

@ncomly-nvidia

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [X] My own task or dataset (give details below)

Reproduction

Not a bug

Expected behavior

Append optional fields in Output Dict

actual behavior

None

additional notes

This is a feature request.

[hello-11]

cc @AdamzNV @ncomly-nvidia @laikhtewari for vis.

[karljang]

@salaki , is this issue still relevant to you? By the way, there has been some updates made so the feature is available in the tensorrt backend:

sampling_params = SamplingParams(
    prompt_logprobs=5,  # Now this will work
    max_tokens=20
)

But, this feature is not yet available in PyTorch workflow 😄

[github-actions[bot]]

Issue has not received an update in over 14 days. Adding stale label.

[github-actions[bot]]

This issue was closed because it has been 14 days without activity since it has been marked as stale.