toolhive
toolhive copied to clipboard
Add usage metrics for vMCP product telemetry
Summary
Implement product telemetry for vMCP to track usage patterns and inform product decisions.
Background
ToolHive has existing patterns for product telemetry:
- Operator telemetry in
pkg/operator/telemetry/for instance tracking - Version checking and update notifications
- ConfigMap-based instance ID persistence
vMCP needs similar product telemetry to understand:
- How many vMCP instances are deployed
- What configurations are commonly used
- Tool call patterns and volumes
Reusable Patterns
| Pattern | Location | Description |
|---|---|---|
| Instance Tracking | pkg/operator/telemetry/telemetry.go |
Instance ID generation |
| Update Checking | pkg/updates/ |
Periodic telemetry collection |
Implementation
Metrics to Track
Deployment Metrics:
- Number of vMCP instances
- Backends per vMCP (count distribution)
- Conflict resolution strategies used (prefix/priority/manual)
- Auth types configured (OIDC, anonymous, token exchange)
Usage Metrics:
- Tool calls per vMCP instance (aggregated)
- Composite tools defined and executed
- Backend distribution (which backends are most used)
Operational Metrics:
- vMCP uptime
- Configuration changes over time
Implementation Steps
-
Define telemetry data model
- What data points to collect
- Privacy considerations (no PII, no tool arguments)
-
Add collection mechanism
- Periodic aggregation of usage data
- Instance identification (anonymous ID)
-
Add reporting endpoint
- Optional opt-in telemetry reporting
- Clear documentation of what's collected
Privacy Considerations
- No personally identifiable information (PII)
- No tool arguments or resource content
- No authentication tokens or secrets
- Aggregated counts only (not individual requests)
- Clear opt-out mechanism
Acceptance Criteria
- [ ] Usage metrics defined and documented
- [ ] Collection mechanism implemented
- [ ] Privacy-preserving (no PII, aggregated data only)
- [ ] Opt-out mechanism available
- [ ] Documentation of what's collected
- [ ] Kubernetes and CLI deployment support