rooooc

Results 2 comments of rooooc

can't start server with small --max-total-tokens. But works fine with big stting

> Hello @rooooc! > > Your issue probably relates to not setting `max-batch-total-tokens` (https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/launcher#maxbatchtotaltokens). By setting different values for `max-total-tokens` and `max-batch-prefill-tokens` you are not controlling the max tokens that...

can't start server with small --max-total-tokens. But works fine with big stting

> Hello @rooooc! > > Your issue probably relates to not setting `max-batch-total-tokens` (https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/launcher#maxbatchtotaltokens). By setting different values for `max-total-tokens` and `max-batch-prefill-tokens` you are not controlling the max tokens that...