akeru icon indicating copy to clipboard operation
akeru copied to clipboard

Support streaming as part of thread runs and LLM generations

Open multipletwigs opened this issue 10 months ago • 0 comments

Problem Statement

  1. Based on a previous PR #9, we have managed to introduce the concept of thread runs, where we await for a response based on the content of the thread.
  2. We should have the option to stream the answer back to the consumer of the api to accommodate for the slow response time of LLMs.
  3. While the adapters are currently generators, we have not supposed streaming over the internet connection yet.

multipletwigs avatar Mar 30 '24 03:03 multipletwigs