semantic-conventions icon indicating copy to clipboard operation
semantic-conventions copied to clipboard

GenAI: Clarify what `gen_ai.request.top_k` means

Open lmolkova opened this issue 1 year ago • 0 comments

OpenAI top_logprobs:

An integer between 0 and 20 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.

Cohere k:

Ensures only the top k most likely tokens are considered for generation at each step. Defaults to 0, min value of 0, max value of 500.

Anthropic top_k:

Only sample from the top K options for each subsequent token. Used to remove "long tail" low probability responses. Learn more technical details here.

We should document what this parameter is, currently it's not clear

https://github.com/open-telemetry/semantic-conventions/blob/26e3779ba8367c7f4c1d5c6e23000cdb51455d86/docs/gen-ai/gen-ai-spans.md?plain=1#L62

lmolkova avatar Sep 09 '24 17:09 lmolkova