LinJianping

[Bug] RuntimeError: CUDA error: operation not permitted when stream is capturing

15

### Checklist - [ ] 1. I have searched related issues but cannot get the expected help. - [ ] 2. The bug has not been fixed in the latest...

Randomly distribute traffic across multiple workers of the same model

9

### Feature request / 功能建议 I have deployed one supervisor and two qwen2-vl-7b-instruct workers. However, I've noticed that currently clients can only query by model_id. I would like to query...