Return more than just text reason for errors.

[michalwarda]

When using llmchain when there is an error while invoking the completion returned the error itself is only a text. This makes it hard to correctly react to those messages and makes it super flaky in case the message changes. ie. when calling ChatOpenAI with too many tokens the result of chain run is:

{:error, _chain, "This model's maximum context length is 16385 tokens. However, your messages resulted in 34966 tokens. Please reduce the length of the messages."}

To react to this specific error type and ie. roll the memory one would need to do some String.contains?("maximum context length") to check if this is the specific error that happened or not. If the error text changes in the future for whatever reason our implementation will start failing. OpenAI returns the type of the error as a separate type field with value context_length_exceeded which is much easier to interpret later in code. Currently there is no way to retrieve this value from the reason returned from run.

Suggested solution.

We can change the API to return error as ie. struct with "message" and optionally "code" or w/e. This is obviously a breaking change.

Another one is to return stringified JSON as the reason ie. for ChatOpenAI and let the user decide on which parts of error they want to use to interpret the error. This is also a semi-breaking change. It does not break the contract but the behavior is changed.

The last one I can think of is to pass custom callbacks for each of the error types to handle but I feel this is a pretty universal non provider dependent problem and still for the sync non streamed calling it would be cool to have the whole error as part of the returned value.

[brainlid]

Hi @michalwarda! Yes, I knew this day was coming. :slightly_smiling_face:

That error message you gave as an example actually comes from OpenAI. It's their text. And yes, they have changed it in the past. I've seen text errors about system outages too.

The only solid errors I've seen consistently are:

• content violation types (ie. violence, hate, ...)
• length

But then you have Llama2 or most OpenAI models that don't give a content violation reason, the assistant response is just, "I can't do that" or something.

I agree that a structured error would be better. I like it. I'm currently in the "breaking change" mode with RCs as I fix/update/rework things.

I'm currently NOT returning the content violation messages from Google, etc. and that would make doing that easier too.

Maybe it's time. :slightly_smiling_face:

Would you like to help by suggesting an error structure that includes everything you're thinking?

[brainlid]

PR #194 changes LLMChain and ChatModels to return an error struct instead of just a string message.

However, there still isn't any specific error detection for this situation.

test/chat_models/chat_open_ai_test.exs:912 is a test that causes this error. Do you want to take a look at detecting this specific situation and now returning a smarter error?