text-generation-webui
text-generation-webui copied to clipboard
Feature Request: Response streaming with HTTP API
This PR extends API extension with /api/v1/generate_stream
endpoint. This endpoint takes same input as /api/v1/generate
, but responds with plain text while outputting tokens as soon as they are ready.
This would allow 3rd party scripts and applications to stream tokens wihtout having to use websocket and guessing gradio function IDs.
It would also allow for simple way to use api from curl and shellscripts:
https://asciinema.org/a/LXzNgmMbdLEv2mg7aCbLJkbMf