[BUG] The health status of chat api server cannot be queried when the chat api server is generating responses.

Open tjtanaa opened this issue 1 year ago • 1 comments

Describe the bug

The health status of chat api server cannot be queried when the chat api server is generating responses.

Sep 04 '24 05:09 tjtanaa

The issue has been identified: Since the server is launched using a single worker, this means that when the model is streaming or generating results, the worker will be occupied. Any incoming requests to the server will need to be queued until the one and only worker is free.

Welcome more suggestions by replying to this comment.

Sep 25 '24 21:09 tjtanaa