不支持高并发，如何提高并发下响应速度

Open Jay-tian opened this issue 1 year ago • 2 comments

十个字使用GPU（T4卡），一个并发大概要1秒多，两个并发平均响应时间就要2秒到3秒，十个并发平均就要9秒多，GPU利用率只到了40%，有没有办法提高并发

Sep 09 '24 03:09 Jay-tian

同问

Sep 09 '24 03:09 njzfw1024

使用vLLM（实验性）或开多个进程。

Sep 10 '24 15:09 fumiama

请问多线程可以么

Oct 29 '24 06:10 superstring

py很难多线程。

Oct 30 '24 12:10 fumiama

This issue was closed because it has been inactive for 15 days since being marked as stale.

Dec 16 '24 04:12 github-actions[bot]