lightllm icon indicating copy to clipboard operation
lightllm copied to clipboard

Inconsistent Output between LightLLM and Transformers Inference Library

Open Lvjinhong opened this issue 5 months ago • 2 comments

When specifying 'max new tokens', LightLLM's output consistently matches this maximum value. However, Transformers sometimes adjust according to the model itself, resulting in outputs shorter than the specified 'max new tokens'. I believe Transformers is correct in this approach. It's implausible to always generate output exactly matching the maximum 'max new tokens' value, as this would only lead to repetitive outputs.

Lvjinhong avatar Jan 19 '24 07:01 Lvjinhong