[BUG] Azure OpenAI > 1.0.0-beta.8 (starting from beta 9) causes JSON streaming payload issues

[timostark]

Describe the bug After the change to azure-json in beta 9 there are assertions when using function calling with streaming. Everything is still working, but ofc the console errors are not OK.

Exception or Stack Trace ChatCompletionsToolCall.java public static ChatCompletionsToolCall fromJson(JsonReader jsonReader) throws IOException

--> throws (>20 times)

2024-07-16T12:08:10.913+02:00 ERROR 33620 --- [oundedElastic-1] c.a.c.i.MethodHandleReflectiveInvoker    : Unexpected end-of-input in field name at [Source: (byte[])"{"choices":[{"content_filter_results":{},"delta":{"tool_calls":[{"funct"; line: 1, column: 72]
2024-07-16T12:08:10.913+02:00 ERROR 33620 --- [oundedElastic-1] c.a.c.i.s.DefaultJsonSerializer          : com.azure.json.implementation.jackson.core.io.JsonEOFException: Unexpected end-of-input in field name at [Source: (byte[])"{"choices":[{"content_filter_results":{},"delta":{"tool_calls":[{"funct"; line: 1, column: 72]

To Reproduce Steps to reproduce the behavior - Use the code snippet below from the official documentation

Code Snippet

	ChatCompletionsOptions optionsPure = new ChatCompletionsOptions(List.of(new ChatRequestUserMessage("What is " +
			"the " +
			"weather at the Tokio?")));

	var func = new FunctionDefinition("getCurrentWeather");
	func.setDescription("Get the current weather for a location");
	func.setParameters(BinaryData.fromString("""
			{"$schema":"https://json-schema.org/draft/2020-12/schema","description":"Weather API request","type":"object","properties":{"location":{"type":"string","description":"The city and state e.g. San Francisco, CA"},"unit":{"type":"string","enum":["C","F"],"description":"Temperature unit"}},"required":["location","unit"]}
			"""));

	optionsPure.setTools(List.of(
			new ChatCompletionsFunctionToolDefinition(func)));
	optionsPure.setStream(true);

	this.openAIClient.getChatCompletionsStream(options.getModel(), optionsPure)
			.forEach(chatCompletions -> {
				if (CoreUtils.isNullOrEmpty(chatCompletions.getChoices())) {
					return;
				}
				ChatResponseMessage delta = chatCompletions.getChoices().get(0).getDelta();
				if (delta.getRole() != null) {
					System.out.println("Role = " + delta.getRole());
				}
				if (delta.getContent() != null) {
					String content = delta.getContent();
					System.out.print(content);
				}
			});

Expected behavior No error messages,,

[junan-trustarc]

Same issue for me as well, functions are working but just dirty error logs. can you fix it

[timostark]

issue is reproducible using your own example (StreamingChatSample.java).

[joshfree]

@alzimmermsft please follow up on this azure-json error in OpenAI

[alzimmermsft]

Thanks for filing this issue @timostark.

@mssfang / @jpalvarezl this appears to be a bug in OpenAIServerSentEvents and how they're being processed. I see a few possible issues here:

• (Not part of this bug) There's an assumption made that the ByteBuffer will have a backing array (ByteBuffer.array()), if the ByteBuffer is native memory this will be false and will result in an exception being thrown.
• (The problem) The stream of ByteBuffers emitted may not cleanly break on a server sent event. I found the cases where the exception is thrown that there is a partial server sent event trailing in the emitted data. Ex, data: {"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":, which is invalid JSON leading to the exception being seen in this issue. We'll need richer checks to make sure that a fully-formed server sent event is being processed.
• (General comment) Don't use 0xA and 0xD to perform checks. While they are correct, many people won't know what these are without researching them (new line characters), please compare against \n and \r so someone looking at this understands what's going on right away.

Here's a code sample to reproduce this without needing to call Azure OpenAI:

Flux<ByteBuffer> source = Flux.fromIterable(Arrays.asList(
    ByteBuffer.wrap("data: {\"choices\":[{\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"}".getBytes(StandardCharsets.UTF_8)),
    ByteBuffer.wrap(",\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}},\"delta\":{\"content\":\" par\"},\"finish_reason\":null,\"index\":0,\"logprobs\":null}],\"created\":1724446441,\"id\":\"id\",\"model\":\"model\",\"object\":\"chat.completion.chunk\",\"system_fingerprint\":\"fingerprint\"}".getBytes(StandardCharsets.UTF_8))));
OpenAIServerSentEvents parser = new OpenAIServerSentEvents(source, ChatCompletions.class);

parser.getEvents().blockLast();

[alzimmermsft]

Forgot to add, when running the reproduction sample, while I didn't see exceptions being thrown anywhere, they are logged (as mentioned by this issue). Add an environment variable of AZURE_LOG_LEVEL=debug to see these logs happening without needing to add an SLF4J logging implementation.

[williamspindox]

Any news?

[maxhov]

@alzimmermsft any news on this issue?

[alzimmermsft]

@mssfang, @jpalvarezl, could you provide @maxhov an update on where this issue currently stands

[maxhov]

@mssfang @jpalvarezl @alzimmermsft could you indicate if this is going to be addressed anytime soon?

[mssfang]

@maxhov Sorry for the late reply. We are going to look into it. Due to other higher priority tasks. It might be slow, but let me see if we can address it before December.

[arey]

I have the same issue with the 1.0.0-beta.11 version provided by LangChain4j.

[devUndef1ned]

I'd say it's quite critical issue for users of the SDK. We use streaming via open azure SDK chat completions. That's why logs of our service are highly polluted by deserialization errors. For us it's a big issue to analyze error logs if they contain that high amount of errors

[Reasch]

When will this bug be fixed? Now because of these error log pollution, I have to downgrade to 1.0.0-beta.7. I hope it will be fixed as soon as possible, and I also want to use the new feature