deepset-ai
feat: add generator of StreamingHandler into pipeline output.
Is your feature request related to a problem? Please describe. We are using the haystack to setup a streaming chatbot demo with LLM. But haystack doesn't have the streaming mode RestAPI. We found that the streaming mode RestAPI needs a streaming generator to get the tokens from handler.
Describe the solution you'd like
- Add the streaming mode parameters in the promptNode run function, support the dynamically enabling and disabling.
- Add the TextIteratorStreamer object into pipeline output and run the pipe() of prompt node in new thread. It can simplify the implementation of streaming RestAPI and support the non-blocked mode.
Describe alternatives you've considered Implement a stremer handler(customer Iterator object) in rest_api and pass it in pipeline.run. It needs to run entire pipeline in other thread or coroutine.
Additional context Add any other context or screenshots about the feature request here.
@yuanwu2017 @julian-risch I suggest we don't close this feature request but add it to the REST API task for V2 as a reminder that users ask for it
@vblagoje sounds fair 👍 Will be good to have this use case in mind once we re-design the REST API.
Won't fix in 1.x, let's use https://github.com/deepset-ai/hayhooks/issues/9 to track the feature in 2.x