vllm icon indicating copy to clipboard operation
vllm copied to clipboard
verified publisher icon vllm-project

higher latency than TGI

Open gravitywp opened this issue 2 years ago • 0 comments

Is it normal to have higher latency than TGI with a low concurrency, such as 1 or 4?

gravitywp avatar Jul 03 '23 03:07 gravitywp