Fix mistral empty choices mistral on azure

[jujuforce]

Background

Mistral model hosted on Azure Foundry send in their first chunk of a chat completion request an empty choices array. For example :

{
  "choices": [],
  "created": 0,
  "id": "",
  "model": "",
  "object": "",
  "prompt_filter_results": [
    {
      "prompt_index": 0,
      "content_filter_results": {
        "hate": {
          "filtered": false,
          "severity": "safe"
        },
        "jailbreak": {
          "filtered": false,
          "detected": false
        },
        "self_harm": {
          "filtered": false,
          "severity": "safe"
        },
        "sexual": {
          "filtered": false,
          "severity": "safe"
        },
        "violence": {
          "filtered": false,
          "severity": "safe"
        }
      }
    }
  ]
}

Summary

I added the necessary handling for this case.

Checklist

• [ ] Tests have been added / updated (for bug fixes / features)
• [ ] Documentation has been added / updated (for bug fixes / features)
• [ ] A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
• [x] I have reviewed this pull request (self-review)

[gr2m]

could this be similar to https://github.com/vercel/ai/pull/10019, where the first chunk comes from model-router?

[jujuforce]

could this be similar to #10019, where the first chunk comes from model-router?

From what I can see, it could pretty much be linked somehow. The weird thing being that I'm 100% I was targetting this model directly : https://ai.azure.com/catalog/models/mistral-medium-2505 So no model routing should be involved but I guess we never know at the end of the day. The fixes in this PR did allow me to request the model no problem. How do you recon we proceed? Thanks

[gr2m]

Can you please share a full usage example? I assume you use @ai-sdk/mistral but set the base URL to your Azure deployment? It feels odd to add azure-specific logic into the mistral provider code 🤔

[jujuforce]

Can you please share a full usage example? I assume you use @ai-sdk/mistral but set the base URL to your Azure deployment? It feels odd to add azure-specific logic into the mistral provider code 🤔

Sure thing.

import { createMistral } from '@ai-sdk/mistral'

const mistralAzureProvider = createMistral({
        fetch: async (input, init?: RequestInit) => {
             //REDACTED
        },
        apiKey: process.env.AZURE_API_KEY,
        baseURL: process.env.AZURE_BASE_URL,
        headers: {
            'REDACTED': 'REDACTED',
        }
    })
    
const mistralAzureModel = mistralAzureProvider('mistral-medium-2505')

const toolLoopAgent = new ToolLoopAgent({
        model: mistralAzureModel,
        tools: tools,
        instructions: systemPrompt('2024', currentDate),
        stopWhen: [stepCountIs(10)],
        output: Output.object({ schema: CustomResponseZodSchema }),
        maxOutputTokens: 10000,
        activeTools: ['REDACTED'],
    })

    const stream = await toolLoopAgent.stream({
        messages: [
            {
                role: 'user',
                content: 'REDACTED',
            },
        ],
    })
    for await (const text of stream.textStream) {
        process.stdout.write(text)
    }
    console.log('\n')

In essence, this is what I’ve been doing: using tool from a MCP server, a complex system prompt, multi-steps function calling, and streamed structured output, with models hosted mainly on Azure and VertexAI deployments. Everything is working great thanks to your AWESOME SDK, except for the Mistral models.

I also tried using createAzure from @ai-sdk/azure for the Mistral models, but I ran into even more issues, such as not being able to easily disable the SDK’s automatic append usage config in the request body, and getting an error out of the model. I did a quick hack to get further, only to have the SDK fail to parse the chunks generated by the models. So I went back to createMistral, managed to find a quick fix to get everything working, and then opened a PR to you guys.

Not sure if this is really Azure specific code, the fix is only handling an empty "choices" array.

I hope this was detailed enough. If not, please let me know.

[jujuforce]

Hi there, Little update, I ended up switching to openai-compatible provider, works fine for Mistral models hosted on Azure for my current needs. This is clearly only a workaround though.

[gr2m]

I'm glad you are unblocked for now.

I don't want to merge the current changes, I'd like to do some more research. I think the current changes patch over the root cause which is Azure adding an extra chunk to the beginning of the stream that is not related to the selected model. I assume the properties do exist, but in this particular case not on the first chunk. Maybe I'm wrong, but want to dig deep first. I just can't prioritize that work right now.

If you or someone else would like to look into it, my first step would be to create examples/ai-core/src/stream-text/azure-mistral.ts, then record the raw chunks and use them in a test in packages/mistral. See https://github.com/vercel/ai/blob/c20c5ff0d7d277ba07fb1b418a569ccd62549199/contributing/testing.md#streamtext for how we do that.

With a test in place that is using the real raw chunks we can look into what the best fix would be, with a high confidence that we won't break anything.