SSE stream chunks are split arbitrarily

[nietsmmar]

Describe the bug When receiving SSE they sometimes are not completely received in one chunk but split into two (or maybe more?).

To Reproduce Send SSE from Edge-Function or other backend endpoint like this:

var chunkJson = {
  token: token,
  finish: finish
};

controller.enqueue(new TextEncoder().encode(JSON.stringify(chunkJson)));

Receive SSE like this:

final res = await supabase.functions
    .invoke('sse', body: {'input': 'sample text'});

(res.data as ByteStream)
    .transform(const Utf8Decoder())
    .listen((val) {
      print(val);
});

Expected behavior Receive chunks like this: {"token":" gest","finish":false}

Actual behavior I do sometimes receive chunks like this:

{"token":" gest","finish":fal

And then in the next received Event is this: se}

Version (please complete the following information): On Linux/macOS

[zhangheng2022]

I am facing the same issue. What are the steps to resolve it?

[nietsmmar]

@zhangheng2022 At the moment I am always trying to parse my chunk as a json. If that does not work I wait for more chunks to add them to my String to try again. This works at the moment. But this is not fail proof. So I'd appreciate some real fix too.

[nietsmmar]

This still happens. Is there any plan on fixing this issue?

[zhangheng2022]

Looking forward to solving this problem.

[nietsmmar]

This is still an issue. Will this be addressed?

[grdsdev]

@Vinzent03 any idea on this?

[Vinzent03]

I tried to reproduce this, but it worked perfectly fine for me. Since you got the messages as stream, I suspect you set the right content-type header, right? (text/event-stream). This is a variant I tried in my testing:

const msg = {
  "token": " gest",
  "finish": false,
};

Deno.serve((_) => {
  let timerId: number | undefined;

  const body = new ReadableStream({
    start(controller) {
      timerId = setInterval(() => {
        controller.enqueue(new TextEncoder().encode(JSON.stringify(msg)));
      }, 1000);
    },
    cancel() {
      if (typeof timerId === "number") {
        clearInterval(timerId);
      }
    },
  });

  return new Response(body, {
    headers: {
      "Content-Type": "text/event-stream",
    },
  });
});

The only issue I could think of is that SSE uses newline characters as separation between events, but it still worked fine in my testing. I might have tested that wrongly, though.

[nietsmmar]

@Vinzent03 Sorry I was super busy and did not have time to check it yet. I will try to set up a simple test example in the next days.

[grdsdev]

Hi @nietsmmar were you able to set up the reproducible example?

All our tests worked fine, let me know if this is still an issue for you, otherwise I'm going to close it.

[grdsdev]

Closing due no response, if that is still an issue for you, feel free to reopen it and send the reproducible example over.

Thanks.

[nietsmmar]

@grdsdev sadly this issue still persists. The problem is, there is no reproducible example I could find yet. It just happens "sometimes" which makes it very hard to debug. The problem is, that when it happens my clients just receive complete gibberish instead of a proper answer from gpt (which is for what I use the stream) as many of the chunks just get missing as I can't deserialize them into their jsons anymore.

[nietsmmar]

Any update on this? @grdsdev This makes it impossible to reliably use stream.

When I have some ChatGPT stream that I 'forward' like this with my edge-function this happens all the time. Most times I am able to stitch the broken events together later but sometimes my workaround fails and my clients just get sentences full of words with missing elements.