feat: Add comprehensive Python testing infrastructure with Poetry

[llbbl]

Add Python Testing Infrastructure

Summary

This PR sets up a comprehensive testing infrastructure for the JetMoE project using Poetry as the package manager and pytest as the testing framework. The setup provides a ready-to-use testing environment where developers can immediately start writing tests.

Changes Made

Package Management

• Poetry Setup: Created pyproject.toml with complete Poetry configuration
• Dependency Migration: Migrated existing dependencies from setup.py to Poetry
• Python Version: Updated to Python 3.10.10+ to meet scattermoe requirements

Testing Dependencies

Added as development dependencies:

• pytest - Main testing framework
• pytest-cov - Coverage reporting with HTML and XML output
• pytest-mock - Mocking utilities for unit tests

Testing Configuration

Configured in pyproject.toml:

• pytest settings:
  • Test discovery patterns for test_*.py and *_test.py
  • Coverage reporting with multiple formats (terminal, HTML, XML)
  • Custom markers: unit, integration, slow
  • Strict mode with verbose output
• Coverage settings:
  • Source tracking for jetmoe package
  • Exclusions for test files and common patterns
  • Coverage threshold set to 0% (should be increased to 80% once tests are written)

Project Structure

Created testing directory structure:

tests/
├── __init__.py
├── conftest.py          # Shared fixtures
├── test_setup_validation.py  # Validation tests
├── unit/
│   └── __init__.py
└── integration/
    └── __init__.py

Fixtures (conftest.py)

Comprehensive fixtures for testing:

• temp_dir - Temporary directory management
• mock_config - Mock JetMoE configuration
• mock_model - Mock model object
• sample_tensor - Sample PyTorch tensors
• sample_input_ids - Sample input IDs
• sample_attention_mask - Sample attention masks
• reset_torch_seed - Reproducible random seeds
• mock_transformers_model - Mock transformers model
• cleanup_cache - Cache directory cleanup
• disable_gpu - Force CPU usage for tests

Development Experience

• Added Poetry script commands: poetry run test and poetry run tests
• Updated .gitignore with comprehensive testing and development entries
• Created validation tests to verify the setup works correctly

Usage Instructions

Initial Setup

# Install dependencies
poetry install

# Run validation tests
poetry run pytest tests/test_setup_validation.py -v --no-cov

Running Tests

# Run all tests
poetry run test

# Run with specific markers
poetry run pytest -m unit
poetry run pytest -m integration
poetry run pytest -m "not slow"

# Run with coverage
poetry run pytest --cov=jetmoe

# Run specific test files
poetry run pytest tests/unit/test_example.py

Writing Tests

1. Place unit tests in tests/unit/
2. Place integration tests in tests/integration/
3. Use fixtures from conftest.py for common test needs
4. Mark tests appropriately: @pytest.mark.unit, @pytest.mark.integration, @pytest.mark.slow

Notes

• Coverage threshold is currently set to 0% to allow the infrastructure to be merged. This should be updated to 80% once actual tests are written.
• The testing infrastructure is designed to be immediately usable - developers can start writing tests right away.
• All test dependencies are properly isolated as development dependencies, not affecting production installations.

Next Steps

1. Write unit tests for core modules (modeling_jetmoe.py, configuration_jetmoe.py, etc.)
2. Write integration tests for end-to-end functionality
3. Update coverage threshold to 80% once tests are in place
4. Consider adding additional testing tools as needed (e.g., pytest-asyncio, pytest-benchmark)