Iman Tabrizian issues

Repositories
Issues
Comments

Results 22 issues of


                                            Iman Tabrizian

Add back 24.05 response sending path

Before: ``` "infbench_summary": { "Requests": 1000, "Requests/s": 56.781948247187536, "E2E Latency(s)": 17.61123087300075, "Avg Latency(s)": 16.54096719381794, "Total Input Tokens": 127000, "Total Output Tokens": 128000, "Avg. Input Tokens / Request": 127.0, "Avg. Output...

Cancel the requests in streaming case by default when closing the connection

Match the behaviour of non-streaming case.