Error when using streaming chat completion

[Ali-Kalout]

Hi all, we sometimes get the following error, when using the chat completion feature while having the stream field as true.

[adampash]

Just to expand on this a little (I work with @Ali-Kalout :) )—it's not easy to tell what the malformed JSON looks like, but it seems possible that this PR could improve the situation.

I also noticed yesterday that OpenAI has added a JSON mode to chat completions. Adding response_format to the completion API call seems like it could help a lot here.

[adampash]

If I'm not mistaken, if I just update my chat_completion params, whatever params aren't Instructor params will be passed through to the API request. Is that right? So the following should work?

    Instructor.chat_completion(
      model: @model,
      response_model: response_model,
      messages: messages,
      stream: stream,
      max_retries: @max_retries,
      # the important part below
      response_format: %{ type: "json_object"}
    )

[petrus-jvrensburg]

I have this PR that helps with debugging: https://github.com/thmsmlr/instructor_ex/pull/46. If you IO.inspect the request before it goes to the LLM, you should be able to see if the param is included in the way you'd expect.

[joechiarella]

I'm also running into this same issue. I believe its happening because of the use of Jason.decode inside of Req's into option. Req is passing blocks of raw text to the function, and those blocks of text don't always split up into parseable chunks of json. I'm unsure at this point if OpenAI is splitting these responses into multiple packets like this or if something in between is buffering the responses.

[thmsmlr]

This is due to how the stream parsing was set up in Instructor initially. The new version will print out the entire text stream if it fails to parse. This is why you're seeing this error. It's because you were calling OpenAI with some incorrect configuration in stream mode, and it was returning an error that the instructor client wasn't parsing. It was just swallowing it.

Solved in the latest release, {:instructor, "~> 0.1.0"}