A. T. Allen comments

Repositories
Issues
Comments

Results 1 comments of


                                            A. T. Allen

[Qwen/Qwen2.5-14B-Instruct-GPTQ-Int8] Bad Responses with High Concurrent Requests

I am having similar issue with TGI. Inference works great when I process single example at a time in SageMaker endpoint but passing multiple requests to handle load testing responses...