text-generation-inference icon indicating copy to clipboard operation
text-generation-inference copied to clipboard

The settings of top_k, typical_p, do_sample in the request do not affect the generation?

Open boxiaowave opened this issue 10 months ago • 0 comments

https://github.com/huggingface/text-generation-inference/blob/ed72e9212620d4de10fbe476f0b7af2ab94e4cd7/router/src/server.rs#L1053

according to the code here, since the GenerateParameters only use the req.temperature, req.frequency_penalty and req.top_logprobs, it seems that we can not set the top_k, do_sample through request with the following parameters: data = { "model": "model", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 1990?"} ], "temperature":1.0, "top_p":1.0, "top_k":0, "typical_p":1.0, "do_sample":False, }

Is this normal or bug?

boxiaowave avatar Apr 23 '24 09:04 boxiaowave