LLM-based evaluators stoping conditions and valid JSON

[davidsbatista]

Within the context of the the LLM-based evaluators, the replies given by OpenAI contain different values for finish_reason:

for instance, content filter was triggered:

The completion for index 0 has been truncated due to the content filter.
    {
        "statements": [
            "There is a major force uniting Americas

A full list of possible stopping conditions is here:

• https://platform.openai.com/docs/guides/text-generation/chat-completions-response-format

• We need to make sure we handle every possible stopping conditions and return a valid JSON

• We can test this by making several successful evaluation runs over all the questions/queries on a dataset