toolhive
toolhive copied to clipboard
Split vMCP e2e tests into e2e and integration test layers
Summary
Reorganize vMCP tests into distinct e2e and integration test layers for faster feedback loops and better edge case coverage.
Background
Current vMCP test architecture:
- ~5,500 lines of e2e tests in
test/e2e/thv-operator/virtualmcp/ - Tests cover: auth discovery, aggregation, filtering, overrides, conflict resolution, composite workflows
- All tests run against a real Kubernetes cluster
- E2E tests are slow due to full cluster setup/teardown
Current Test Files
test/e2e/thv-operator/virtualmcp/
├── suite_test.go (101 lines)
├── helpers.go (631 lines)
├── virtualmcp_auth_discovery_test.go (1,591 lines)
├── virtualmcp_aggregation_filtering_test.go (322 lines)
├── virtualmcp_aggregation_overrides_test.go (295 lines)
├── virtualmcp_conflict_resolution_test.go (686 lines)
├── virtualmcp_composite_parallel_test.go (374 lines)
├── virtualmcp_composite_sequential_test.go (317 lines)
├── virtualmcp_discovered_mode_test.go (391 lines)
├── virtualmcp_inline_auth_test.go (467 lines)
└── virtualmcp_yardstick_base_test.go (337 lines)
Problem
- Slow CI feedback - Full e2e tests take significant time
- Edge case difficulty - Hard to test timeouts, network failures, partial failures in e2e
- No test layer separation - All tests require full Kubernetes environment
Proposed Structure
Integration Tests (test/integration/vmcp/)
Fast tests that don't require Kubernetes:
- Aggregator logic (conflict resolution, capability merging)
- Router logic (tool routing, backend selection)
- Composer logic (workflow execution, template expansion)
- Config loading and validation
- Mock backend interactions
E2E Tests (test/e2e/thv-operator/virtualmcp/)
Full cluster tests for end-to-end validation:
- CRD creation and reconciliation
- Full request flow (client → vMCP → backend)
- Auth flows (OIDC, token exchange)
- Status reporting
Implementation
-
Create integration test structure
test/integration/vmcp/ ├── aggregator_test.go # Conflict resolution, capability merging ├── router_test.go # Request routing logic ├── composer_test.go # Workflow execution ├── config_test.go # Configuration validation └── helpers.go # Mock backends, test utilities -
Extract testable logic
- Ensure vMCP packages are unit-testable
- Add interfaces for backend interactions (already exists)
- Create mock implementations for testing
-
Move edge case tests to integration
- Timeout handling
- Partial backend failures
- Invalid configurations
- Concurrent request handling
-
Update CI pipeline
- Run integration tests on every PR
- Run e2e tests on merge to main (or nightly)
Acceptance Criteria
- [ ] Integration test directory created
- [ ] Core logic tests extracted from e2e
- [ ] Integration tests run without Kubernetes
- [ ] Edge case tests (timeouts, failures) in integration layer
- [ ] E2E tests remain for full flow validation
- [ ] CI pipeline updated for two-tier testing
- [ ] Test documentation updated