toolhive icon indicating copy to clipboard operation
toolhive copied to clipboard

Split vMCP e2e tests into e2e and integration test layers

Open JAORMX opened this issue 3 weeks ago • 0 comments

Summary

Reorganize vMCP tests into distinct e2e and integration test layers for faster feedback loops and better edge case coverage.

Background

Current vMCP test architecture:

  • ~5,500 lines of e2e tests in test/e2e/thv-operator/virtualmcp/
  • Tests cover: auth discovery, aggregation, filtering, overrides, conflict resolution, composite workflows
  • All tests run against a real Kubernetes cluster
  • E2E tests are slow due to full cluster setup/teardown

Current Test Files

test/e2e/thv-operator/virtualmcp/
├── suite_test.go                              (101 lines)
├── helpers.go                                 (631 lines)
├── virtualmcp_auth_discovery_test.go          (1,591 lines)
├── virtualmcp_aggregation_filtering_test.go   (322 lines)
├── virtualmcp_aggregation_overrides_test.go   (295 lines)
├── virtualmcp_conflict_resolution_test.go     (686 lines)
├── virtualmcp_composite_parallel_test.go      (374 lines)
├── virtualmcp_composite_sequential_test.go    (317 lines)
├── virtualmcp_discovered_mode_test.go         (391 lines)
├── virtualmcp_inline_auth_test.go             (467 lines)
└── virtualmcp_yardstick_base_test.go          (337 lines)

Problem

  1. Slow CI feedback - Full e2e tests take significant time
  2. Edge case difficulty - Hard to test timeouts, network failures, partial failures in e2e
  3. No test layer separation - All tests require full Kubernetes environment

Proposed Structure

Integration Tests (test/integration/vmcp/)

Fast tests that don't require Kubernetes:

  • Aggregator logic (conflict resolution, capability merging)
  • Router logic (tool routing, backend selection)
  • Composer logic (workflow execution, template expansion)
  • Config loading and validation
  • Mock backend interactions

E2E Tests (test/e2e/thv-operator/virtualmcp/)

Full cluster tests for end-to-end validation:

  • CRD creation and reconciliation
  • Full request flow (client → vMCP → backend)
  • Auth flows (OIDC, token exchange)
  • Status reporting

Implementation

  1. Create integration test structure

    test/integration/vmcp/
    ├── aggregator_test.go      # Conflict resolution, capability merging
    ├── router_test.go          # Request routing logic
    ├── composer_test.go        # Workflow execution
    ├── config_test.go          # Configuration validation
    └── helpers.go              # Mock backends, test utilities
    
  2. Extract testable logic

    • Ensure vMCP packages are unit-testable
    • Add interfaces for backend interactions (already exists)
    • Create mock implementations for testing
  3. Move edge case tests to integration

    • Timeout handling
    • Partial backend failures
    • Invalid configurations
    • Concurrent request handling
  4. Update CI pipeline

    • Run integration tests on every PR
    • Run e2e tests on merge to main (or nightly)

Acceptance Criteria

  • [ ] Integration test directory created
  • [ ] Core logic tests extracted from e2e
  • [ ] Integration tests run without Kubernetes
  • [ ] Edge case tests (timeouts, failures) in integration layer
  • [ ] E2E tests remain for full flow validation
  • [ ] CI pipeline updated for two-tier testing
  • [ ] Test documentation updated

JAORMX avatar Dec 02 '25 20:12 JAORMX