text-generation-webui icon indicating copy to clipboard operation
text-generation-webui copied to clipboard

Feature Request: Response streaming with HTTP API

Open im-not-tom opened this issue 1 year ago • 0 comments

This PR extends API extension with /api/v1/generate_stream endpoint. This endpoint takes same input as /api/v1/generate, but responds with plain text while outputting tokens as soon as they are ready.

This would allow 3rd party scripts and applications to stream tokens wihtout having to use websocket and guessing gradio function IDs.

It would also allow for simple way to use api from curl and shellscripts:

https://asciinema.org/a/LXzNgmMbdLEv2mg7aCbLJkbMf

im-not-tom avatar Apr 23 '23 11:04 im-not-tom