.Net: Bug: When AgentOrchestration.InvokeAsync returns OrchestrationResult of type ChatMessageContent the metadata of message content is null

[dimable]

Describe the bug I tired to override HandoffOrchestration<TInput, TOutput> and GroupChatOrchestration<TInput, TOutput> classes in order to have typed input and output <string, ChatMessageContent>. In the orchestartor callback ResponseCallback the input param of type ChatMessageContent contains Metadata dictionary (message Id and other key-pairs). But when orchestration.InvokeAsync returns result (of ChatMessageContent), it contains only Content property value, Metadata property is null.

To Reproduce Steps to reproduce the behavior:

1. Implement custom orchestrator like

public class AgentHandoffOrchestrator(ChatHistory chatHistory, OrchestrationHandoffs handoffs, params Agent[] agents) : HandoffOrchestration<string, ChatMessageContent>(handoffs, agents)
{
    protected override ValueTask StartAsync(IAgentRuntime runtime, TopicId topic, IEnumerable<ChatMessageContent> input, AgentType? entryAgent)
    {
        chatHistory.AddRange(input);
        return base.StartAsync(runtime, topic, chatHistory, entryAgent);
    }
}

2. Create an instance of the custom orchestrator

var orchestration = new AgentHandoffOrchestrator(history, OrchestrationHandoffs.StartWith(agent)
   .Add(agent, software, chart),
   //.Add(agent, software, "Transfer if the query is related to software.")
   //.Add(agent, chart, "Transfer to this agent to create a chart to visualize data."),
   agent, software, chart)
{
    LoggerFactory = loggerFactory,
    ResponseCallback = ResponseCallback,
};

ChatMessageContent content = null;
InProcessRuntime runtime = new();
await runtime.StartAsync(cancellationToken);
var result = await orchestration.InvokeAsync(prompt, runtime, cancellationToken);
content = await result.GetValueAsync(TimeSpan.FromSeconds(300), cancellationToken); // content.Metadata is null
await runtime.RunUntilIdleAsync();

private ValueTask ResponseCallback(ChatMessageContent response)
{
   // response.Metadata is NOT null
    _logger.LogInformation("# {AuthorName}\n\t{Content}", response.AuthorName, response.Content);
    return ValueTask.CompletedTask;
}

3. In ResponseCallback method response.Metadata is not null, but content.Metadata returned from result.GetValueAsync... is null

Platform

• Language: C#
• Source: "Microsoft.SemanticKernel.Agents.Core" Version="1.60.0" "Microsoft.SemanticKernel.Agents.Orchestration" Version="1.60.0-preview" "Microsoft.SemanticKernel.Agents.Runtime.InProcess" Version="1.60.0-preview"
• AI model: OpenAI:GPT-4o
• IDE: Visual Studio 2022
• OS: Windows 11

[crickman]

@TaoChenOSU - This seems to be similiar to https://github.com/microsoft/semantic-kernel/issues/12547. Can you please include in your analysis?

[TaoChenOSU]

Hi @dimable,

Thank you for your question!

What you observed is expected as the result from the Handoff orchestration is not an agent response.

In Handoff, when the agent that is currently handling the conversation decides to end the conversation, it calls a function with a summary of the conversation, as shown here: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Agents/Orchestration/Handoff/HandoffActor.cs#L183. The orchestration packages the summarization as a ChatMessageContent and return it as the result of the orchestration. That particular message doesn't contain any metadata.

If you are looking for token usage or other metadata, you need to capture the agent responses via the ResponseCallback. You then can find a function call message that the agent issues to terminate the orchestration.

[dimable]

Oh, then it makes sense. I needed only message Id, so I made use of ResponseCallback 🙂 and doing LIFO in the handler, when runtime RunUntilIdleAsync returns I pop that last message Id from the stack. Currently works as expected. Thank you Tao for looking into it 🤝

[dimable]

@TaoChenOSU still there is one question... If the final response is not from an agent, but from end_task_with_summary function, then why content.Content contains actual answer to my prompt instead of empty string? So that means the agent didn't end the conversation yet, which means with the response should come message Id (in metadata)

...
ChatMessageContent content = await result.GetValueAsync(TimeSpan.FromSeconds(300), cancellationToken); 
...

EndAsync is invoked at every response received from LLM, but passes only text part futher.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity.