nano-vllm

nano-vllm copied to clipboard

Can I use the CPU for inference?

1

I want to use the CPU for inference. Can it work? Is it possible to not install flash-attn?

beausoft

Mo-Love

* [ADD] Add TTFT, TPOT metrics. * [FIX] remove unnecessary dependencies. * [FIX] replace 'is False' to 'not'. * [FIX] modified some prompts. * [FIX] trailing blanks. * [Fix] redundant...

mumupika