inference icon indicating copy to clipboard operation
inference copied to clipboard

Published 20 hours ago • verified publisher icon xorbitsai

xinference并发处理问题

Open ccly1996 opened this issue 9 months ago • 0 comments

Describe the bug

我用fastGPT接入了xinference部署的vllm qwen32b，测试并发的时候会遇到跑4个并发的时候xinference后台报错，然后ui里也看不到跑的模型了，显卡还在100%占用

To Reproduce

To help us to reproduce this bug, please provide information below:

  1. Your Python version.3.10
  2. The version of xinference you use.0.11.0
  3. Versions of crucial packages.
  4. Full stack of the error.
  5. Minimized code to reproduce the error.

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Add any other context about the problem here. vllm 0.4.1 2d91f7e8605f6672144f6fda2d51023a 9c93967387dda3c5aa4bc807fd0c5c05 8c95b152839255140bda960d1d5e2117

ccly1996 avatar May 15 '24 15:05 ccly1996