Task 7: Integration Tests

[slin1237]

Task 7: Integration Tests

Overview

Create comprehensive integration tests to validate the end-to-end PVC storage support functionality.

Scope

• Unit tests for individual components
• Integration tests for the complete flow
• E2E tests for real cluster validation
• Performance and stress tests

Files to Create/Modify

• pkg/utils/storage/storage_test.go - Enhance existing tests
• pkg/modelagent/gopher_test.go - Add PVC test cases
• pkg/controller/v1beta1/basemodel/controller_test.go - Job creation tests
• test/e2e/pvc_storage_test.go - New E2E test file
• cmd/ome-agent/model_metadata_agent_test.go - Agent unit tests

Test Implementation

Unit Tests for Storage Package

Enhance pkg/utils/storage/storage_test.go with test cases for:

• Valid PVC URI parsing with various subpath formats
• Invalid URI formats and error handling
• Storage type detection for PVC URIs
• Edge cases like empty URIs, missing components, trailing slashes

Model Agent Tests

Add to pkg/modelagent/gopher_test.go test cases for:

• PVC exists and is bound scenario
• PVC not found error handling
• PVC not bound (pending) status
• Invalid PVC URI format
• Proper status updates (MetadataPending, Failed)
• Mock Kubernetes client interactions

BaseModel Controller Tests

Add to pkg/controller/v1beta1/basemodel/controller_test.go test cases for:

• Metadata job creation when BaseModel has PVC storage
• Job spec validation (correct image, args, volumes)
• Job monitoring and status updates
• Handling job success and failure scenarios
• PVC validation before job creation
• Idempotency (job already exists)
• Controller reconciliation loops

E2E Tests

Create test/e2e/pvc_storage_test.go with comprehensive end-to-end tests:

TestPVCStorageEndToEnd:

• Create PVC and wait for it to be bound
• Populate PVC with model files (including config.json)
• Create BaseModel with PVC storage URI
• Verify metadata extraction job runs successfully
• Verify BaseModel metadata is populated
• Create InferenceService using the PVC model
• Verify pod has PVC mounted correctly
• Check volume mounts and subpaths

TestPVCStorageAccessModes:

• Test ReadWriteOnce (RWO) PVC behavior
• Test ReadWriteMany (RWX) PVC behavior
• Verify scheduling constraints are handled correctly

TestPVCStorageErrorCases:

• Test non-existent PVC handling
• Test unbound PVC behavior
• Test invalid subpath scenarios
• Test permission errors and RBAC issues

Model Metadata Agent Tests

Create cmd/ome-agent/model_metadata_agent_test.go with tests for:

• Successful config.json extraction from model directory
• Handling missing config.json files
• Parsing various config.json formats (HuggingFace, custom)
• Metadata field mapping and validation
• BaseModel update via Kubernetes API
• Error handling for malformed JSON
• Permission errors when reading files

Test Matrix

Component	Test Type	Coverage Target
Storage Package	Unit	100%
Model Agent	Unit	PVC handling paths
BaseModel Controller	Unit	Job creation logic
Model Metadata Agent	Unit	Config extraction
End-to-End	Integration	Complete flow
Performance	Load	100 concurrent PVCs

Performance Tests

BenchmarkPVCVerification:

• Benchmark PVC verification speed
• Measure URI parsing performance
• Test concurrent PVC checks

TestPVCStorageScale:

• Test with 100+ BaseModels using PVCs
• Verify no resource leaks
• Check job cleanup and TTL
• Monitor controller performance under load

Acceptance Criteria

• [ ] Unit test coverage > 80% for new code
• [ ] All integration tests pass in CI
• [ ] E2E tests validate complete flow
• [ ] Performance tests show no regression
• [ ] Error cases properly handled and tested
• [ ] Tests are maintainable and well-documented

Test Environment Setup

• Kind cluster with CSI driver for PVC tests
• Pre-populated PVCs with sample models
• Mock model files with valid config.json

Dependencies

• All implementation tasks completed
• Test infrastructure available
• Sample model files prepared

Estimated Effort

5-6 hours

[kris-gaudel]

Hi is this still available?