.Net: Bug: Azure OpenAI client fails with 401 when throttling

[dluc]

When using AzureOpenAITextEmbeddingGenerationService SK client and sending too many requests, the Azure service throttling leads to a "401 Unauthorized" error instead of "429 Too Many Requests".

• SK version: 1.20.0
• .NET version: 8.0.401

I inspected the HTTP requests and noticed that the internal retry logic is sending a malformed request during retries. It appears to be adding the Authorization header twice (with the same token), e.g.

• Request 1 .. 7: ok
• Request 8: fails with HTTP Status Code 429
• Request 8 is retried with two auth tokens: fails with HTTP Status Code 401

Code to repro:

public static class Program
{
    public static async Task Main()
    {
        var client = new AzureOpenAITextEmbeddingGenerationService(
            endpoint: "https://....openai.azure.com/",
            deploymentName: "text-embedding-ada-002",
            credential: new DefaultAzureCredential()
        );

        for (int i = 0; i < 200; i++)
        {
            Console.WriteLine($"## {i}");
            await client.GenerateEmbeddingsAsync([RndStr(), RndStr(), RndStr(), RndStr(), RndStr(), RndStr(), RndStr()]);
        }
    }

    private static string RndStr()
    {
        var random = new Random();
        return new(Enumerable.Repeat("ABCDEF GHIJKLMNOP QRSTUVW XYZabcdefgh ijklmnopqrst uvwxyz0123 456789", 8000)
            .Select(s => s[random.Next(s.Length)]).ToArray());
    }
}

Output:

## 0
## 1
## 2
## 3
## 4
## 5
## 6
## 7
## 8
Unhandled exception. Microsoft.SemanticKernel.HttpOperationException: Service request failed.
Status: 401 (Unauthorized)

 ---> System.ClientModel.ClientResultException: Service request failed.
Status: 401 (Unauthorized)

   at Azure.AI.OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
   at Azure.AI.OpenAI.Embeddings.AzureEmbeddingClient.GenerateEmbeddingsAsync(BinaryContent content, RequestOptions options)
   at OpenAI.Embeddings.EmbeddingClient.GenerateEmbeddingsAsync(IEnumerable`1 inputs, EmbeddingGenerationOptions options, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.RunRequestAsync[T](Func`1 request)
   --- End of inner exception stack trace ---
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.RunRequestAsync[T](Func`1 request)
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetEmbeddingsAsync(String targetModel, IList`1 data, Kernel kernel, Nullable`1 dimensions, CancellationToken cancellationToken)
   at Program.Main() in Program.cs:line 21
   at Program.<Main>()

[dluc]

looks like a bug in the underlying Azure.AI.OpenAI 2.0.0-beta.5, I reported it here https://github.com/Azure/azure-sdk-for-net/issues/46109 including a workaround

[markwallace-microsoft]

@RogerBarreto I'm assigning this to you, can you monitor the issue Devis created an pick up the fix when it becomes available.

[rogerbarreto]

The original issue was closed, setting this as complete as the original issue is addressed and will be available automatically to Semantic Kernel package in a future release.

• https://github.com/Azure/azure-sdk-for-net/issues/46109

[dluc]

@RogerBarreto the outstanding open issue here is the reliance on a pre-release package with a noticeable (quite annoying) bug that impacts all SK consumers.

Rather than waiting for the Azure SDK fix to propagate to SK and SK fix to propagate to consumers, I'd suggest that SK includes a workaround as soon as possible, e.g. by injecting a delegating handler:

internal sealed class AuthFixHandler : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        if (request.Headers.TryGetValues("Authorization", out var headers) && headers.Count() > 1)
        {
            request.Headers.Authorization = new AuthenticationHeaderValue(
                request.Headers.Authorization!.Scheme,
                request.Headers.Authorization.Parameter);
        }

        return base.SendAsync(request, cancellationToken);
    }
}