Performance of RESTful API

[kihanc]

I have measured the performance of Mistral 7B w/ and w/o Restful API, and got 7.1 reqs/s and 0.48 reqs/s individually. (avg. input/output tokens: 232/208, A100 80GB)

[mrwyattii]

Hi @kihanc we are aware of performance issues related to the restful API and will be providing an update on this feature soon. Thanks

[mrwyattii]

@kihanc please try with the latest main branch. I have made improvements that allow us to match performance the RESTful API to our Python API (see #328).

[BaiMoHan]

@kihanc please try with the latest main branch. I have made improvements that allow us to match performance the RESTful API to our Python API (see #328).

Hey, how can I make Restful start and listen on 0.0.0.0 instead of localhost? I often encounter situations where I deploy my application on a remote machine but cannot test it using external API tools like Postman because it only listens on localhost.

[PawanOsman]

@kihanc please try with the latest main branch. I have made improvements that allow us to match performance the RESTful API to our Python API (see #328).

Hey, how can I make Restful start and listen on 0.0.0.0 instead of localhost? I often encounter situations where I deploy my application on a remote machine but cannot test it using external API tools like Postman because it only listens on localhost.

I think it's using localhost by default and there is no option to set it, you can modify this file locally until they add an option to set the hostname

[BaiMoHan]

@kihanc please try with the latest main branch. I have made improvements that allow us to match performance the RESTful API to our Python API (see #328).

Hey, how can I make Restful start and listen on 0.0.0.0 instead of localhost? I often encounter situations where I deploy my application on a remote machine but cannot test it using external API tools like Postman because it only listens on localhost.

I think it's using localhost by default and there is no option to set it, you can modify this file locally until they add an option to set the hostname

Thank you!

[kihanc]

@mrwyattii Thanks for the quick update! Your test code works perfectly on our end. But, if we hit more than 100 requests at once, things get wonky and connections are reset. It would be awesome if you could add support for concurrent connections on your API server.

[mrwyattii]

@kihanc if you increase the number of restful_processes (which defaults to 32) does that help? Perhaps try 64?

mii.serve(model_name, restful_processes=64, enable_restful_api=True)

[kihanc]

@mrwyattii I already tried that before but that was not helpful.

[IKACE]

Do we have any updates on this issue? I am also observing "Connection reset" failures when request rate is high