kihanc issues

Repositories
Issues
Comments

Results 1 issues of


                                            kihanc

Performance of RESTful API

I have measured the performance of Mistral 7B w/ and w/o Restful API, and got 7.1 reqs/s and 0.48 reqs/s individually. (avg. input/output tokens: 232/208, A100 80GB)