OpenLLM icon indicating copy to clipboard operation
OpenLLM copied to clipboard

Published 20 hours ago • verified publisher icon bentoml

feat: Stream completions

Open Nanguage opened this issue 2 years ago • 1 comments

Feature request

Is it possible to support stream completion similar to the OpenAI API?

Motivation

Can save users waiting time and improve user experience.

Other

No response

Nanguage avatar Jun 13 '23 13:06 Nanguage

This has to do with SSE support on BentoML. There is a feature roadmap currently working on in the community discord.

aarnphm avatar Jun 13 '23 14:06 aarnphm

This is now finished and supported

aarnphm avatar Sep 05 '23 01:09 aarnphm