[Feature Request]: CRT-Based Semantic Tool Router for Dynamic MCP Servers

[pbarbas]

🧭 Epic

Title: CRT-Based Semantic Tool Router Plugin for Dynamic MCP Servers
Goal: Introduce a new plugin (mcpgateway/plugins/crt_router) that enables the MCP Gateway to dynamically select and expose only the most semantically relevant tools based on the user’s prompt — without relying on LLM calls. This predictive router leverages a learned latent difficulty model (via Chinese Remainder Theorem fusion) to replace or complement existing LLM-based tool selection.

---

🙋‍♂️ User Story 1

As a: developer using the MCP ecosystem
I want: the gateway to semantically filter and display only relevant tools for a given prompt
So that: the LLM isn’t overloaded by irrelevant tools and delivers faster, higher-quality responses.

✅ Acceptance Criteria

• Given a user prompt is received
  When the CRT Router computes semantic relevance bins
  Then only the top-K or below-threshold tools are exposed to the LLM context.
• Given a registry with unrelated tools
  When the prompt has no semantic connection to them
  Then those tools are excluded from the final catalog.

---

🙋‍♂️ User Story 2

As a: model integrator or AI engineer
I want: to reduce tool overload during dynamic server resolution
So that: inference latency and prompt length both decrease, and reasoning accuracy improves.

✅ Acceptance Criteria

• Given many tools registered in the MCP Gateway
  When only the semantically relevant subset is injected into the prompt
  Then tool invocation reliability improves and cost/latency are reduced.

---

🔧 Implementation Sketch

Implementation Mode: Plugin

• Implement as a plugin under mcpgateway/plugins/crt_router/
• Register a new routing mode "crt" at startup:

  def register_plugin(app):
      from mcpgateway.plugins.crt_router.semantic_router import CRTRouter
      app.state.router_modes["crt"] = CRTRouter()
  

• Core logic in semantic_router.py:
  • Load calibration artifacts (difficulty bins S, prime list, prior π(s), and calibrated success tables âₘ(s)).
  • Compute modular residues → fuse posteriors → compute expected loss → rank tools.
• Add config toggle:

  MCPGATEWAY_ROUTER_MODE=crt
  

• Extend /dynamic/{id}/tools endpoint:

  /dynamic/{id}/tools?router=crt&k=10&threshold=0.72
  

• Optional endpoint:

  /router/health
  

  → returns model calibration checksum, entropy, and version info.
• Store calibration JSON under /data/calibration/ or /examples/.

---

🧩 Dependencies

• Dynamic Server API (/dynamic) — #123
• Taggable Tool Metadata — #586 / #676
• (Optional) LLM Router Mode parity — #130

---

🧠 Design Rationale

Current Dynamic Server selection supports rule-based and LLM-based filtering.
However, LLM selection:

• incurs extra inference cost,
• introduces nondeterminism, and
• lacks transparent calibration.

The CRT Router is predictive, not generative.
It learns semantic “difficulty” and tool-relevance structure offline, then applies a modular Bayesian fusion (via CRT) to choose tools in microseconds — achieving deterministic, explainable selection.

---

📈 Expected Business Value

• Efficiency: avoids unnecessary LLM calls and shortens context windows.
• Predictability: deterministic behavior; can run in restricted or air-gapped deployments.
• Scalability: same architecture extends to hundreds of tools with no performance penalty.
• Auditability: route decisions can be explained using posterior entropy or expected loss metrics.

In short: CRT makes dynamic MCP servers intelligent selectors, not just LLM-driven filters.

---

👤 Ownership

Assignee: @pbarbas Reviewers: @crivetimihai

---

🗓️ Milestone

Target: Release 1.0.0

---

✅ Definition of Done

• [ ] Plugin module under mcpgateway/plugins/crt_router/ registered successfully.
• [ ] CRT inference engine loads calibration and computes relevance deterministically.
• [ ] /dynamic/{id}/tools?router=crt returns filtered catalog consistent with top-K semantic bins.
• [ ] Config and environment toggle fully functional (MCPGATEWAY_ROUTER_MODE).
• [ ] 90 %+ test coverage for new code paths.
• [ ] Admin UI lists “CRT Router” as a selectable routing strategy.
• [ ] Documentation updated with usage, configuration, and calibration steps.
• [ ] Prometheus metrics: router_crt_requests_total, latency, and cache hit rate.

---

🧠 References

• Blueprint: CRT Routing v2 – Robust, Information-Adaptive Router
• Related Works: FrugalGPT (Chen et al., 2024), Fusion of Experts (2024), Context Rot (Hong et al., 2025)
• Goal: Replace heuristic LLM filtering with predictive, modular semantic routing.