djl-serving icon indicating copy to clipboard operation
djl-serving copied to clipboard

Published 20 hours ago • verified publisher icon deepjavalibrary

Add multi-token support

Open ymwangg opened this issue 9 months ago • 0 comments

Description

Per-token text is not supported. This may needs some changes on vllm/lmi_dist side. Per-token cum_logprob is not supported, but should be easy to add.

ymwangg avatar May 08 '24 17:05 ymwangg