helixml
when the UI connects to a livestream session, prefix can be truncated
when the UI connects to a live streaming session (one where the model is still generating output), it only receives new updates, often missing out the start.
This is particularly noticable now that NATS has made the output damn quick
Ideally, when you open a websocket to start streaming responses, you start by receiving in a single chunk all the result so far for a given interaction. Will we need to start buffering this in the API server?
we could probably either do nats streaming or keep updating the database with deltas while it's generating every 3-5 seconds and then load from the DB
Would NATS streaming solve this with stream level persistence?
I like that more because with the 3-5 second option matching up the end of the database text with the start of the streaming text is tricky
Cool let's plug it in
lukemarsden.net @.*** @lmarsden https://twitter.com/lmarsden
On Sun, 4 Feb 2024 at 15:58, Karolis Rusenas @.***> wrote:
yeah I think it would
— Reply to this email directly, view it on GitHub https://github.com/helixml/helix/issues/166#issuecomment-1925804044, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACATUVAXMAY4HA6LWM6NUTYR6V3BAVCNFSM6AAAAABCYVTSKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVHAYDIMBUGQ . You are receiving this because you authored the thread.Message ID: @.***>
Maybe there's also a quicker workaround to get the frontend to subscribe to the websocket faster when a new session is created?