sglang icon indicating copy to clipboard operation
sglang copied to clipboard
Published 5 months ago • verified publisher icon sgl-project

Max Length.

Open srush opened this issue 2 years ago • 2 comments

Is there anyway to truncate text based on tokens? I really like that as a user I don't need to think about tokens. But to save memory I would like something like s += left_trunc(inp, 500) that keeps it reasonably sized.

srush avatar Jan 17 '24 21:01 srush

After this line s += left_trunc(inp, 500), do you mean inp <= 500 or s + inp <= 500? Either one should be easy to implement!

Ying1123 avatar Jan 18 '24 10:01 Ying1123

I was thinking inp <=500 tokens.

srush avatar Jan 18 '24 16:01 srush

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

github-actions[bot] avatar Jul 25 '24 06:07 github-actions[bot]