JSON failure w/gpt-4-0125-preview.

[lukehollenback]

When I configure AI Code Reviewer to run against gpt-4-0125-preview (a turbo model), I get the following failure (sometimes on repeat, sometimes only once) in the AI Code Reviewer step every single time it runs →

Error: SyntaxError: Unexpected token ` in JSON at position 0
    at JSON.parse (<anonymous>)
    at /home/runner/work/_actions/freeedcom/ai-codereviewer/main/webpack:/open-ai-reviewer/lib/main.js:151:1
    at Generator.next (<anonymous>)
    at fulfilled (/home/runner/work/_actions/freeedcom/ai-codereviewer/main/webpack:/open-ai-reviewer/lib/main.js:28:1)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

This failure results in no comments being made on the pull request.

Making it run against gpt-4-1106-preview instead resolves the issue and the action works well.

[villesau]

I think that's probably an issue on OpenAI end generating valid JSON unfortunately. If you have insight on what is the actual return value, that would help!

[lukehollenback]

It looks like the action forces JSON Mode for the gpt-4-1106-preview model, but not for the gpt-4-0125-preview one. I bet the simplest fix would be to add || OPENAI_API_MODEL === "gpt-4-0125-preview" || OPENAI_API_MODEL === "gpt-4-turbo-preview" to → https://github.com/freeedcom/ai-codereviewer/blob/a9a064dfa1db8c83f40ef63f6e247fa09c935ed6/src/main.ts#L130.

[lukehollenback]

As an aside, what is the simplest way to debug and tell you what the actual return value is?

[lukehollenback]

I just put up https://github.com/freeedcom/ai-codereviewer/pull/57, which seems likely to fix this issue. Very simple enhancement.

[SvenBunge]

Used gpt-4-turbo-preview that points to gpt-4-0125-preview and still receive a bunch of errors:

Error: SyntaxError: Unexpected token ` in JSON at position 0 at JSON.parse () at /home/runner/work/_actions/SvenBunge/ai-codereviewer/main/webpack:/open-ai-reviewer/lib/main.js:151:1 at Generator.next () at fulfilled (/home/runner/work/_actions/SvenBunge/ai-codereviewer/main/webpack:/open-ai-reviewer/lib/main.js:28:1) at processTicksAndRejections (node:internal/process/task_queues:96:5)

[lukehollenback]

Used gpt-4-turbo-preview that points to gpt-4-0125-preview and still receive a bunch of errors:

Error: SyntaxError: Unexpected token ` in JSON at position 0 at JSON.parse () at /home/runner/work/_actions/SvenBunge/ai-codereviewer/main/webpack:/open-ai-reviewer/lib/main.js:151:1 at Generator.next () at fulfilled (/home/runner/work/_actions/SvenBunge/ai-codereviewer/main/webpack:/open-ai-reviewer/lib/main.js:28:1) at processTicksAndRejections (node:internal/process/task_queues:96:5)

Yeah, because until #57 gets merged, this action only forces JSON mode with the 1106 model - not with any other models.

Even with #57, I get occasional errors with 0125 - but nowhere near as many. I suspect this is happens on larger diffs because of a large number of tokens that are being generated and cutoff - so I also submitted #61 to make max generated tokens configurable.

[gleb-altarey]

I am seeing this on gpt-3.5-turbo. Is it even supported? The workflow stage is marked as a success.

[Wzixiao]

Perhaps we can also extract the first "{" and its last "}" and parse it? (In the past, when there was no json_object parameter, this was my solution. the common situation is that the parsing fails due to the existence of "code blocks".)

Additionally, there might be situations where there are not enough tokens. We could possibly use a retry mechanism where the output from the current request is used as input for a new request.