toolhive icon indicating copy to clipboard operation
toolhive copied to clipboard

Add usage metrics for vMCP product telemetry

Open JAORMX opened this issue 3 weeks ago • 0 comments

Summary

Implement product telemetry for vMCP to track usage patterns and inform product decisions.

Background

ToolHive has existing patterns for product telemetry:

  • Operator telemetry in pkg/operator/telemetry/ for instance tracking
  • Version checking and update notifications
  • ConfigMap-based instance ID persistence

vMCP needs similar product telemetry to understand:

  • How many vMCP instances are deployed
  • What configurations are commonly used
  • Tool call patterns and volumes

Reusable Patterns

Pattern Location Description
Instance Tracking pkg/operator/telemetry/telemetry.go Instance ID generation
Update Checking pkg/updates/ Periodic telemetry collection

Implementation

Metrics to Track

Deployment Metrics:

  • Number of vMCP instances
  • Backends per vMCP (count distribution)
  • Conflict resolution strategies used (prefix/priority/manual)
  • Auth types configured (OIDC, anonymous, token exchange)

Usage Metrics:

  • Tool calls per vMCP instance (aggregated)
  • Composite tools defined and executed
  • Backend distribution (which backends are most used)

Operational Metrics:

  • vMCP uptime
  • Configuration changes over time

Implementation Steps

  1. Define telemetry data model

    • What data points to collect
    • Privacy considerations (no PII, no tool arguments)
  2. Add collection mechanism

    • Periodic aggregation of usage data
    • Instance identification (anonymous ID)
  3. Add reporting endpoint

    • Optional opt-in telemetry reporting
    • Clear documentation of what's collected

Privacy Considerations

  • No personally identifiable information (PII)
  • No tool arguments or resource content
  • No authentication tokens or secrets
  • Aggregated counts only (not individual requests)
  • Clear opt-out mechanism

Acceptance Criteria

  • [ ] Usage metrics defined and documented
  • [ ] Collection mechanism implemented
  • [ ] Privacy-preserving (no PII, aggregated data only)
  • [ ] Opt-out mechanism available
  • [ ] Documentation of what's collected
  • [ ] Kubernetes and CLI deployment support

JAORMX avatar Dec 02 '25 20:12 JAORMX