.Net: Bug: AI Agent Service, grounding with Bing throws "Object reference not set to an instance of an object."

[davidames]

Describe the bug When making a request to an agent via the AI Agent Service that is grounded with Bing, InvokeStreamingAsync() throws an exception after the last message is rendered. This appears to be related to the citation.

To Reproduce

Test Code

    [Fact]
    [Experimental("OPENAI001")]
    public async Task InternetSearchFlattened()
    {
        var foundryConfig = GetService<AiFoundryConfig>();
        var client = AzureAIAgent.CreateAzureAIClient(foundryConfig.ConnectionString, new DefaultAzureCredential());
        var agentsClient = client.GetAgentsClient();
        var getAgentResponse =  await agentsClient.GetAgentAsync(AgentId);

        AzureAIAgent  agent = new(getAgentResponse.Value, agentsClient);
        AgentThread thread = new AzureAIAgentThread(agentsClient);
        
        await foreach (var responseUpdate in agent.InvokeStreamingAsync("What are today's news headlines", thread))
        {
            Logger.Debug("Streaming chunk: {Chunk}", responseUpdate.Message.Content);
        }
    }

Exception

System.NullReferenceException
Object reference not set to an instance of an object.
   at Azure.AI.Projects.TextAnnotationUpdate.get_TextToReplace()
   at Microsoft.SemanticKernel.Agents.AzureAI.Internal.AgentThreadActions.GenerateStreamingAnnotationContent(TextAnnotationUpdate annotation)
   at Microsoft.SemanticKernel.Agents.AzureAI.Internal.AgentThreadActions.GenerateStreamingMessageContent(String assistantName, MessageContentUpdate update)
   at Microsoft.SemanticKernel.Agents.AzureAI.Internal.AgentThreadActions.InvokeStreamingAsync(AzureAIAgent agent, AgentsClient client, String threadId, IList`1 messages, AzureAIInvocationOptions invocationOptions, ILogger logger, Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Agents.AzureAI.Internal.AgentThreadActions.InvokeStreamingAsync(AzureAIAgent agent, AgentsClient client, String threadId, IList`1 messages, AzureAIInvocationOptions invocationOptions, ILogger logger, Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Agents.AzureAI.Internal.AgentThreadActions.InvokeStreamingAsync(AzureAIAgent agent, AgentsClient client, String threadId, IList`1 messages, AzureAIInvocationOptions invocationOptions, ILogger logger, Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Microsoft.SemanticKernel.Diagnostics.ActivityExtensions.RunWithActivityAsync[TResult](Func`1 getActivity, Func`1 operation, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Diagnostics.ActivityExtensions.RunWithActivityAsync[TResult](Func`1 getActivity, Func`1 operation, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Diagnostics.ActivityExtensions.RunWithActivityAsync[TResult](Func`1 getActivity, Func`1 operation, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Microsoft.SemanticKernel.Agents.AzureAI.AzureAIAgent.InvokeStreamingAsync(ICollection`1 messages, AgentThread thread, AzureAIAgentInvokeOptions options, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Agents.AzureAI.AzureAIAgent.InvokeStreamingAsync(ICollection`1 messages, AgentThread thread, AzureAIAgentInvokeOptions options, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Agents.AzureAI.AzureAIAgent.InvokeStreamingAsync(ICollection`1 messages, AgentThread thread, AzureAIAgentInvokeOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at SKSpikes.Experiments.AgentChat.Agent004AzureAiServicesAgent.InternetSearchFlattened() in C:\_cfdev\semantic-kernel-spikes\Experiments\AgentChat\Agent004AzureAiServicesAgent.cs:line 208
   at SKSpikes.Experiments.AgentChat.Agent004AzureAiServicesAgent.InternetSearchFlattened() in C:\_cfdev\semantic-kernel-spikes\Experiments\AgentChat\Agent004AzureAiServicesAgent.cs:line 208
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass47_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 259
--- End of stack trace from previous location ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90

Debug Output:

ℹ️ 08:04:50.230 Test "Agent004AzureAiServicesAgent" initialized 🐞 08:04:59.948 Streaming chunk: "Today's" 🐞 08:04:59.949 Streaming chunk: " news" 🐞 08:04:59.949 Streaming chunk: " headlines" 🐞 08:04:59.950 Streaming chunk: " include" 🐞 08:05:00.000 Streaming chunk: ":

" 🐞 08:05:00.126 Streaming chunk: "1" 🐞 08:05:00.126 Streaming chunk: "." 🐞 08:05:00.127 Streaming chunk: " Hamas" 🐞 08:05:00.127 Streaming chunk: " negoti" 🐞 08:05:00.127 Streaming chunk: "ators" 🐞 08:05:00.128 Streaming chunk: " are" 🐞 08:05:00.128 Streaming chunk: " considering" 🐞 08:05:00.128 Streaming chunk: " a" 🐞 08:05:00.128 Streaming chunk: " cease" 🐞 08:05:00.128 Streaming chunk: "fire" 🐞 08:05:00.128 Streaming chunk: " proposal" 🐞 08:05:00.128 Streaming chunk: " from" 🐞 08:05:00.129 Streaming chunk: " Israel" 🐞 08:05:00.150 Streaming chunk: "," 🐞 08:05:00.192 Streaming chunk: " mediated" 🐞 08:05:00.192 Streaming chunk: " by" 🐞 08:05:00.192 Streaming chunk: " Egyptian" 🐞 08:05:00.192 Streaming chunk: " and" 🐞 08:05:00.193 Streaming chunk: " Q" 🐞 08:05:00.193 Streaming chunk: "at" 🐞 08:05:00.253 Streaming chunk: "ari" 🐞 08:05:00.316 Streaming chunk: " representatives" 🐞 08:05:00.316 Streaming chunk: "," 🐞 08:05:00.316 Streaming chunk: " which" 🐞 08:05:00.316 Streaming chunk: " includes" 🐞 08:05:00.316 Streaming chunk: " demands" 🐞 08:05:00.316 Streaming chunk: " for" 🐞 08:05:00.316 Streaming chunk: " Hamas" 🐞 08:05:00.316 Streaming chunk: " to" 🐞 08:05:00.366 Streaming chunk: " dis" 🐞 08:05:00.366 Streaming chunk: "arm" 🐞 08:05:00.459 Streaming chunk: "【3:0†source】"

Expected behavior No exception thrown

Other Information This only occurs with when the Bing search tool is invoked - ie, a request to "Why is the sky blue?" will not throw an exception.

Platform

• Language: C#
• Source: MS.SK.Agents.AzureAI .46.0-preview . & MS.SK 1.40
• AI model: gpt4o (2024-08-06)
• IDE: Rider
• OS: Windows

[davidames]

I found a workaround:

SK was bringing in an implicit package: Azure.AI.Projects 1.0.0-beta.3

I fixed the issue by explicitly referencing: Azure.AI.Projects 1.0.0-beta.6

Leaving issue open for visibility to the SK team

[markwallace-microsoft]

We have updated to the latest package. These packages should be becoming stable soon so I expect this type of issue will be eliminated then.