worker_generate_stream接口在100并发量下响应非常慢

[njzfw1024]

请问有哪里可以改进的地方嘛？

`@app.post('/worker_generate_stream') async def generate_stream(request: Request):

global model_semaphore, global_counter
global_counter += 1
params = await request.json()
if model_semaphore is None:
         model_semaphore = asyncio.Semaphore(args.limit_model_concurrency)
await model_semaphore.acquire()

worker.send_heart_beat()
generator = worker.generate_stream_gate(params)
background_tasks = BackgroundTasks()
background_tasks.add_task(partial(release_model_semaphore, fn=worker.send_heart_beat))
return StreamingResponse(generator, background=background_tasks, media_type="application/json")`

[ErfeiCui]

提高并发响应速度，可以使用vllm或者Imdeploy进行worker部署和适配