Azure OpenAI Client Streaming with Function (tools)

[Grogdunn]

The AzureOpenAI client doesn't support function calling (now called tools in Azure SDK) in streaming response.

See: https://github.com/spring-projects/spring-ai/blob/1e98f82e03529506c9afaa650717b3819633d573/models/spring-ai-azure-openai/src/main/java/org/springframework/ai/azure/openai/AzureOpenAiChatClient.java#L168

To reproduce, create a simple Client with a tool, and invoke, the choice contains the function call, but is ignored.

[Grogdunn]

working on a PR https://github.com/optionfactory/spring-ai/tree/azure-openai-tool-streaming-478

[tzolov]

@Grogdunn , this is true. I've implemented the streaming function calling for Gemini, OpenAI and Mistral AI (there is some common abstractions for last two) but haven't had time to do it for the Azure OpenAI client. Would appreciate your help!

[Grogdunn]

Hi @tzolov ,

I've found another issue, but I don't know how to fix at the moment. https://github.com/spring-projects/spring-ai/blob/1e98f82e03529506c9afaa650717b3819633d573/models/spring-ai-azure-openai/src/main/java/org/springframework/ai/azure/openai/AzureOpenAiChatClient.java#L474

In case of streaming in the line above (in all client Azure, OpenAI, Mistral, Gemini) "broke" the streaming ad do non streaming request. So, in streaming environment, with function call the LLM is invoked as not streaming.

[tzolov]

Resolved with a7eb28ac17debbaaaf227802eb0385f3daea4fd8