Jee Jee Li

icon indicating an email [email protected]

icon location Chengdu, China

Results 248 comments of Jee Jee Li

[Bug]: vllm much slower on long context inputs when using --enable-lora even when lora is not used

> > @badrjd Has this issue been resolved for you? If not, you can try adding `--max-seq-len-to-capture 48000`.It's most likely due to this reason > > is there any explanations?...

[Feature]: vllm Multicar inference bnb model TP is not supported

Indeed not supported, only in-flight quantization supports TP

[Kernel] Add trition.autotune to address the high latency overhead of punica kernels

Thanks for your contribution, have you tested the performance of autotune on models like llama?

[Kernel] Add trition.autotune to address the high latency overhead of punica kernels

@congcongchen123 you can refer to https://github.com/vllm-project/vllm/blob/main/tests/lora/test_llama_tp.py

[Usage]: VLLM Inference - 2x slower with LoRA rank=256 vs none.

We have a benchmark result at [slack_lora_thread]( https://vllm-dev.slack.com/archives/C07V3D6F493/p1739803031668149?thread_ts=1738948465.513199&cid=C07V3D6F493). We are aware of this issue and will optimizing the lora performance. Could you please provide your model and LoRA config?

[Usage]: VLLM Inference - 2x slower with LoRA rank=256 vs none.

I will test #14626 asap, and will provibe the test result here. @rtx-8000 @varun-sundar-rabindranath

[Usage]: VLLM Inference - 2x slower with LoRA rank=256 vs none.

![Image](https://github.com/user-attachments/assets/da167017-a58f-465d-825d-d32bb3d418ff) ![Image](https://github.com/user-attachments/assets/5735cf3a-9b8b-4a82-9859-d794188a0159)

[Usage]: VLLM Inference - 2x slower with LoRA rank=256 vs none.

> So what could be wrong for me to have slower performance for lower rank, especially for max_num_seqs (request rate) = 1 or no such big difference for higher rate?...