text-generation-inference
text-generation-inference copied to clipboard
The settings of top_k, typical_p, do_sample in the request do not affect the generation?
https://github.com/huggingface/text-generation-inference/blob/ed72e9212620d4de10fbe476f0b7af2ab94e4cd7/router/src/server.rs#L1053
according to the code here, since the GenerateParameters only use the req.temperature, req.frequency_penalty and req.top_logprobs, it seems that we can not set the top_k, do_sample through request with the following parameters: data = { "model": "model", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 1990?"} ], "temperature":1.0, "top_p":1.0, "top_k":0, "typical_p":1.0, "do_sample":False, }
Is this normal or bug?