microsoft
[BUG] azureaibestpractices tool got called when azurebestpractices is supposed to be called
Describe the bug
I've noticed that sometimes azureaibestpractices tool got called when azurebestpractices is supposed to be called. We need to find a way to eliminate this confusion for LLM.
Expected behavior
There should be a clear difference on when LLM should call which tool.
Actual behavior
LLM gets confused and call azureaibestpractices tool when it is supposed to call azurebestpractices
Reproduction Steps
Try this prompt "Create a Python Function App with a HTTP Trigger"
Environment
No response
cc/ @XiaofuHuang
- With only Azure MCP enabled, I couldn’t reproduce the issue (tested 10 times).
- The problem appears when GitHub Copilot execute “Optimizing tool selection” (virtual tools), which happens when more than 128 tools are enabled. In that mode, Copilot groups and re-describes tools, which blurs the difference between azurebestpractices and azureaibestpractices. In this setup, I can reproduce azureaibestpractices being chosen instead of azurebestpractices.
We will try to change the namespace description to ensure the grouped/virtual-tool descriptions preserve the distinction between the two namespaces, so the copilot keeps choosing the right tool.
@fanyang-mono We have moved Azure AI best practices into the best practice namespace, but this does not resolve the issue where the best practice tool cannot be called if more than 128 tools are enabled in GitHub Copilot. This is due to GitHub Copilot enabling virtual tools and rewriting the tool set descriptions.
I tested the latest Azure MCP server (2.0.8) and saw get_bestpractices tool being called most of the time with the prompt with Copilot Chat (0.33.5). I am going to close this issue, since the reported issue has been resolved mostly.
There is an issue with the latest Copilot Chat extension (0.35.0). See more details at https://github.com/microsoft/vscode/issues/282803