Json streaming POC (WIP)

Open avaer opened this issue 1 year ago • 1 comments

This is a simple POC of how we could implement streaming output for our chat messages. This is a highly desirable feature and almost all AI inference endpoints support it.

Since we deal exclusively in formatted JSON messages, it's a little tricky to extract the partially completed JSON keys/values before they are done (e.g. printing half of a deep string that's not even terminated).

This POC shows how we can do it using https://www.npmjs.com/package/@streamparser/json-whatwg. Example parse is included.

Not to be merged, this is just a hack for now.

Oct 23 '24 12:10 avaer

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
chat	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 5, 2024 9:53pm
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 5, 2024 9:53pm

Oct 23 '24 12:10 vercel[bot]