mcp icon indicating copy to clipboard operation
mcp copied to clipboard

[Tool Description] Improve Documentation Search tool description for better LLM selection

Open alzimmermsft opened this issue 3 months ago • 1 comments

Problem

The Documentation Search tool description is mostly good but has some inconsistencies and formatting issues that could improve LLM tool selection clarity. Current estimated confidence score is ~0.6 (above target of ≥0.5, but improvable).

Current Description Issues:

  • Missing guidance on when NOT to use this tool
  • Inconsistent description (detailed first part vs generic second part)
  • Poor parameter explanation that doesn't match functionality
  • Formatting issues (missing space after period)
  • Missing limitations about query types and scope

Current Description

Search official Microsoft/Azure documentation to find the most relevant and trustworthy content for a user's query. This tool returns up to 10 high-quality content chunks (each max 500 tokens), extracted from Microsoft Learn and other official sources. Each result includes the article title, URL, and a self-contained content excerpt optimized for fast retrieval and reasoning. Always use this tool to quickly ground your answers in accurate, first-party Microsoft/Azure knowledge.This tool is a hierarchical MCP command router. Sub commands are routed to MCP servers that require specific fields inside the "parameters" object. To invoke a command, set "command" and wrap its args in "parameters". Set "learn=true" to discover available sub commands.

Suggested Improvement

Search official Microsoft/Azure documentation to find the most relevant and trustworthy content for a user's query. This tool returns up to 10 high-quality content chunks (each max 500 tokens), extracted from Microsoft Learn and other official sources. Each result includes the article title, URL, and a self-contained content excerpt optimized for fast retrieval and reasoning. Use this tool when you need authoritative information about Azure services, APIs, best practices, tutorials, or troubleshooting from official Microsoft sources. Do not use this tool for third-party documentation, community forums, or non-Microsoft content - this tool specifically searches Microsoft Learn and official Azure documentation. To invoke this tool, provide your search query in the "question" parameter. The tool works best with specific, technical questions about Azure services and features rather than very broad or general queries.

Acceptance Criteria

  • [ ] Update tool description to include clear usage guidance
  • [ ] Fix formatting issues (missing space)
  • [ ] Remove inconsistent hierarchical router description
  • [ ] Clarify parameter usage (question parameter)
  • [ ] Specify what types of queries work best
  • [ ] Maintain confidence score ≥0.6

Related: Tool description review for improved LLM selection accuracy

alzimmermsft avatar Sep 03 '25 22:09 alzimmermsft

Copied from https://github.com/Azure/azure-mcp/issues/823

alzimmermsft avatar Sep 03 '25 22:09 alzimmermsft