blacker comments

Repositories
Issues
Comments

Results 2 comments of


                                            blacker

[Performance]: Qwen 7b chat model, under 128 concurrency, the CPU utilization rate is 100%, and the GPU SM utilization rate is only about 60%-75%. Is it a CPU bottleneck?

You can start the VLLM API interface service, which will have CPU and GPU utilization, for example Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs,...

unexpected error when creating modelInstanceState: [json.exception.out_of_range.403] key 'name' not found

I meet the same problem also.