[BUG] CompleteChatAsync throws 'response_format value as json_schema is enabled only for api versions 2024-08-01-preview and later'

[andersgranaker]

Library name and version

Azure.AI.OpenAI 2.0.0

Describe the bug

When calling the chat completion endpoint with ChatResponseFormat JsonSchemaFormat a 400 BadRequest error is thrown with the message: response_format value as json_schema is enabled only for api versions 2024-08-01-preview and later'

The model deployment I'm using is GPT4o 2024-08-06. The same code and endpoint works with version Azure.AI.OpenAI Beta-12.

UPDATE: The api version used by the stable version 2.0.0 of this library does not support json_schema - making it in practice a unsupported feature by this version of the lib.

Expected behavior

Call the ChatCompletion endpoint with JsonSchemaFormat as ChatResponseFormat. Use a GPT4o model deployment of the latest version. Expect a 200 response formatted as the json schema specifies.

Actual behavior

Call the ChatCompletion endpoint with JsonSchemaFormat as ChatResponseFormat. Use a GPT4o model deployment of the latest version. A HTTP 400 error is thrown. The error message says to use a later version of the GPT4o deployment, but we are using the latest one available.

Reproduction Steps

Call the ChatCompletion endpoint with JsonSchemaFormat as ChatResponseFormat. Use a GPT4o model deployment of the latest version. This fails with the 2.0.0 version of your package.

Try again with version 2.0.0 beta-12 and see that it works.

Environment

dotnet 8

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jpalvarezl @ralph-msft @trrwilson.

[grankko]

looking at the network traffic: the 2.0.0 beta package uses api-version=2024-08-01-preview the 2.0.0 package uses api-version=2024-06-01

Only one service version is in the enum in AzureOpenAiClientOptions. I guess this basically means that structured outputs are not supported in the 2.0.0 version if this library?

    //
    // Summary:
    //     The version of the service to use.
    public enum ServiceVersion
    {
        V2024_06_01
    }

[joakimriedel]

This is a bummer, just upgraded to 2.0 version and looked forward to use ResponseFormat. Looking at the code, the newer API versions seems to be hidden behind a compiler flag, see source.

    /// <summary> The version of the service to use. </summary>
    public enum ServiceVersion
    {
        V2024_06_01 = 0,
#if !AZURE_OPENAI_GA
        V2024_08_01_Preview = 1,
        V2024_10_01_Preview = 3,
#endif
    }

[trrwilson]

@andersgranaker, @grankko, @joakimriedel -- thanks for getting in touch!

As you discovered, structured outputs are only supported in the recent preview/beta releases of the Azure.AI.OpenAI library. The reason they don't appear in the stable version is that there isn't yet a stable Azure OpenAI service version that includes structured outputs; that 2024-06-01 label for api-version is the latest GA target and that pre-dated structured outputs feature availability. We don't point library releases labeled as stable/GA to service API versions that are still in preview, as it can lead to a lot of compatibility issues and confusion when switching between preview versions can sometimes include breaking changes that would be unacceptable between GA versions.

As soon as there's a new stable service API label that supports the feature, we'll immediately update the stable release of the library to point to it and enable everything that's been newly inducted into GA status. Although we can't comment on any specific ETA for when that'll be just yet, I'll just offhandedly note that the Microsoft Ignite conference starts one month from today and that -- generally! -- teams often like to have contemporary GA service API labels available near the major conferences. Generally.

[grankko]

Thanks for getting back @trrwilson, looking forward to Ignite!

For this issue, maybe a more explicit error message would be good when trying to use JsonSchemaFormat with the 2.0.0 version of the library. It would be consistent with using other features that are still in preview. Example from AzureOpenAIClient:

throw new InvalidOperationException($"The preview Batch feature area is not available in this GA release of the Azure OpenAI Service. To use this capability, please use a preview version of the library.");

[V0v1kkk]

A little workaround:

var azureOptions = new AzureOpenAIClientOptions();
SetCustomVersion(azureOptions, "2024-08-01-preview");
var openAiClient = new AzureOpenAIClient(new Uri(azureEndpoint), new ApiKeyCredential(azureApiKey), azureOptions);
var chatClient = openAIClient.GetChatClient("gpt-4o");

void SetCustomVersion(AzureOpenAIClientOptions options, string customVersion)
{
    Type optionsType = typeof(AzureOpenAIClientOptions);
    FieldInfo versionField = optionsType.GetField("<Version>k__BackingField", BindingFlags.Instance | BindingFlags.NonPublic);
    if (versionField != null)
    {
        versionField.SetValue(options, customVersion);
    }
    else
    {
        throw new InvalidOperationException("Unable to find the Version backing field.");
    }
}

@andersgranaker , @grankko, @joakimriedel

[TrueCodePoet]

I was having this issue today. very frustrating with the o3-mini model. after trying a bunch of different libraries. this fixed my issue Thanks! will be glad for this to stop blocking new models. I just updated with the correct version.

[erictu22]

Hope this helps someone but please check your apiVersion parameter when sending requests to azure OpenAI and make sure it's up to date

https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#rest-api-versioning