Auto-Apply Agent Hyperparameter Recommendations and Expand Coverage

Open ooples opened this issue 2 months ago • 0 comments

Auto-Apply Agent Hyperparameter Recommendations and Expand Coverage

User Story

As a machine learning developer using AI-assisted model building, I want the agent's hyperparameter recommendations to be automatically applied to my model when possible, so that I can benefit from AI-guided optimization without manually configuring every parameter, while still maintaining full control over the configuration process.

Problem Statement

Current State:

The AI agent system (GetAgentRecommendationsAsync) analyzes datasets and provides intelligent recommendations including:

Model type selection (SuggestedModelType)
Hyperparameter values (SuggestedHyperparameters dictionary)
Model selection reasoning (ModelSelectionReasoning)

However, these recommendations are underutilized:

// src/PredictionModelBuilder.cs - Current ApplyAgentRecommendations
private void ApplyAgentRecommendations(AgentRecommendation<T, TInput, TOutput> recommendation)
{
    // Applies model type recommendation (with console guidance only)
    if (_model == null && recommendation.SuggestedModelType.HasValue)
    {
        Console.WriteLine($"The AI agent recommends using: {recommendation.SuggestedModelType.Value}");
        Console.WriteLine($"To use this recommendation, configure your builder:");
        Console.WriteLine($"  builder.UseModel(/* create {recommendation.SuggestedModelType.Value} instance */);");
    }

    // Note: Hyperparameter recommendations are currently stored in recommendation.SuggestedHyperparameters
    // but not auto-applied. Future enhancement: Apply hyperparameters to compatible models.
}

Problems with Current Approach:

Manual Work Required: Users must manually extract and apply hyperparameter values
Lost Optimization Potential: AI recommendations are ignored unless user reads console output
Inconsistent Coverage: Not all model types have hyperparameter recommendation support
No Reflection/Introspection: Can't automatically set properties on models
Complex Model Factory: 80+ model types with different constructor signatures makes auto-creation challenging
Silent Failures: No feedback when hyperparameters can't be applied

Missing Features:

No automatic hyperparameter application to existing models
No hyperparameter introspection API (IConfigurableModel or similar)
No validation that recommended hyperparameters are compatible with model
No reporting of which hyperparameters were/weren't applied
Limited coverage across model types (not all models support agent-recommended hyperparameters)

User Impact Examples:

// Current workflow - manual application required
var result = await builder
    .ConfigureAgentAssistance(options => options.EnableModelSelection())
    .BuildAsync(data, labels);

// User sees: "The AI agent recommends using: RandomForest"
// User sees: "Recommendation details available in result.AgentRecommendation"

// User must then manually do:
var hyperparameters = result.AgentRecommendation.SuggestedHyperparameters;
// How do I apply these? Which properties? What if model doesn't support them?

Proposed Solution

Create a comprehensive hyperparameter application system that:

Auto-applies hyperparameters when a model is already configured via UseModel()
Validates compatibility between recommendations and model capabilities
Reports what was applied successfully and what wasn't
Expands coverage to all ~80+ model types in the library
Maintains user control through opt-in configuration

Design Philosophy

Incremental Enhancement: Start with property-based approach, expand to interfaces later
Explicit Opt-In: Hyperparameter auto-application requires explicit configuration
Safe Defaults: Never override user-specified values, only fill in missing ones
Clear Reporting: User always knows what the agent did and why
Extensibility: Easy to add support for new hyperparameters and models

Architecture

Phase 1: Property-Based Hyperparameter Application (Foundation)

Goal: Apply agent hyperparameter recommendations using reflection on existing models.

AC 1.1: Create HyperparameterApplicator<T> Service (8 points)

Requirements:

[ ] Create src/Agents/HyperparameterApplicator.cs
[ ] Define public class HyperparameterApplicator<T>

[ ] Core method:

public HyperparameterApplicationResult ApplyHyperparameters(
    IFullModel<T> model,
    Dictionary<string, object> hyperparameters,
    bool overrideExistingValues = false)

[ ] Use reflection to discover writable properties on model
[ ] Map hyperparameter names (case-insensitive) to property names
[ ] Support common hyperparameter types: int, double, bool, enum
[ ] Validate type compatibility before setting
[ ] Track success/failure for each hyperparameter

Implementation Details:

public class HyperparameterApplicator<T>
{
    public HyperparameterApplicationResult ApplyHyperparameters(
        IFullModel<T> model,
        Dictionary<string, object> hyperparameters,
        bool overrideExistingValues = false)
    {
        var result = new HyperparameterApplicationResult();

        foreach (var (key, value) in hyperparameters)
        {
            try
            {
                // Find matching property (case-insensitive)
                var property = model.GetType()
                    .GetProperties(BindingFlags.Public | BindingFlags.Instance)
                    .FirstOrDefault(p =>
                        p.Name.Equals(key, StringComparison.OrdinalIgnoreCase) &&
                        p.CanWrite);

                if (property == null)
                {
                    result.AddFailure(key, "Property not found or not writable");
                    continue;
                }

                // Skip if value already set and override not requested
                var currentValue = property.GetValue(model);
                if (currentValue != null &&
                    !IsDefaultValue(currentValue) &&
                    !overrideExistingValues)
                {
                    result.AddSkipped(key, "Property already has non-default value");
                    continue;
                }

                // Convert and validate type
                object convertedValue = ConvertValue(value, property.PropertyType);

                // Set the property
                property.SetValue(model, convertedValue);
                result.AddSuccess(key, convertedValue);
            }
            catch (Exception ex)
            {
                result.AddFailure(key, ex.Message);
            }
        }

        return result;
    }

    private object ConvertValue(object value, Type targetType)
    {
        // Handle enums
        if (targetType.IsEnum && value is string enumString)
        {
            return Enum.Parse(targetType, enumString, ignoreCase: true);
        }

        // Handle nullable types
        var underlyingType = Nullable.GetUnderlyingType(targetType);
        if (underlyingType != null)
        {
            targetType = underlyingType;
        }

        // Standard conversion
        return Convert.ChangeType(value, targetType);
    }

    private bool IsDefaultValue(object value)
    {
        if (value == null) return true;

        var type = value.GetType();
        if (type.IsValueType)
        {
            return value.Equals(Activator.CreateInstance(type));
        }

        return false;
    }
}

public class HyperparameterApplicationResult
{
    public List<(string Name, object Value)> Applied { get; } = new();
    public List<(string Name, string Reason)> Skipped { get; } = new();
    public List<(string Name, string Reason)> Failed { get; } = new();

    public int SuccessCount => Applied.Count;
    public int TotalCount => Applied.Count + Skipped.Count + Failed.Count;
    public bool HasFailures => Failed.Count > 0;

    public void AddSuccess(string name, object value) => Applied.Add((name, value));
    public void AddSkipped(string name, string reason) => Skipped.Add((name, reason));
    public void AddFailure(string name, string reason) => Failed.Add((name, reason));

    public string GetSummary()
    {
        var sb = new StringBuilder();
        sb.AppendLine($"Hyperparameter Application Summary:");
        sb.AppendLine($"  Applied: {Applied.Count}");
        sb.AppendLine($"  Skipped: {Skipped.Count}");
        sb.AppendLine($"  Failed: {Failed.Count}");

        if (Applied.Count > 0)
        {
            sb.AppendLine("  Successfully applied:");
            foreach (var (name, value) in Applied)
            {
                sb.AppendLine($"    - {name} = {value}");
            }
        }

        if (Failed.Count > 0)
        {
            sb.AppendLine("  Failed to apply:");
            foreach (var (name, reason) in Failed)
            {
                sb.AppendLine($"    - {name}: {reason}");
            }
        }

        return sb.ToString();
    }
}

Validation:

[ ] Successfully sets int, double, bool properties
[ ] Successfully sets enum properties from string values
[ ] Handles nullable types correctly
[ ] Respects overrideExistingValues flag
[ ] Provides detailed success/failure reporting
[ ] Case-insensitive property matching works
[ ] Skips read-only properties gracefully

AC 1.2: Integrate HyperparameterApplicator into ApplyAgentRecommendations (5 points)

Requirements:

[ ] Update src/PredictionModelBuilder.cs ApplyAgentRecommendations method
[ ] Apply hyperparameters when _model is already set via UseModel()
[ ] Output detailed application results to console
[ ] Respect new configuration option: AutoApplyHyperparameters (default: true)

Updated ApplyAgentRecommendations:

private void ApplyAgentRecommendations(AgentRecommendation<T, TInput, TOutput> recommendation)
{
    // Apply model type recommendation (existing logic)
    if (_model == null && recommendation.SuggestedModelType.HasValue)
    {
        Console.WriteLine($"\n=== AGENT RECOMMENDATION ===");
        Console.WriteLine($"The AI agent recommends using: {recommendation.SuggestedModelType.Value}");
        // ... existing console output
    }

    // NEW: Apply hyperparameter recommendations
    if (_model != null &&
        recommendation.SuggestedHyperparameters != null &&
        recommendation.SuggestedHyperparameters.Count > 0 &&
        _agentConfiguration?.AutoApplyHyperparameters == true)
    {
        Console.WriteLine($"\n=== APPLYING AGENT HYPERPARAMETER RECOMMENDATIONS ===");

        var applicator = new HyperparameterApplicator<T>();
        var result = applicator.ApplyHyperparameters(
            _model,
            recommendation.SuggestedHyperparameters,
            overrideExistingValues: false);

        Console.WriteLine(result.GetSummary());

        if (result.HasFailures)
        {
            Console.WriteLine("\nSome hyperparameters could not be applied. " +
                            "This may be expected if your model type doesn't support them.");
        }

        Console.WriteLine("====================================================\n");
    }
}

Validation:

[ ] Hyperparameters applied when model is set via UseModel()
[ ] No application when AutoApplyHyperparameters = false
[ ] Console output shows exactly what was applied
[ ] User can see which hyperparameters failed and why

AC 1.3: Add AutoApplyHyperparameters to AgentConfiguration (2 points)

Requirements:

[ ] Add public bool AutoApplyHyperparameters { get; set; } = true; to AgentConfiguration

[ ] Add fluent configuration method to AgentConfigurationBuilder:

public AgentConfigurationBuilder EnableHyperparameterApplication(bool enable = true)
{
    _autoApplyHyperparameters = enable;
    return this;
}

Usage Example:

var result = await builder
    .UseModel(new RandomForest<double>())
    .ConfigureAgentAssistance(options => options
        .EnableModelSelection()
        .EnableHyperparameterApplication()) // NEW
    .BuildAsync(data, labels);

// Agent will now auto-apply hyperparameters to the RandomForest model

Phase 2: Expand Hyperparameter Coverage (High Impact)

Goal: Ensure all major model types have comprehensive hyperparameter recommendation support.

AC 2.1: Audit Current Hyperparameter Coverage (3 points)

Requirements:

[ ] Review GetAgentRecommendationsAsync to identify which hyperparameters are currently recommended
[ ] Audit all ~80 model types to identify:
- Which have configurable hyperparameters
- Which hyperparameters are most impactful for each model family
- Which hyperparameters are currently missing from agent recommendations
[ ] Create coverage matrix documenting:
- Model type
- Available hyperparameters
- Currently recommended by agent (yes/no)
- Priority for recommendation support

Deliverable: HYPERPARAMETER_COVERAGE_MATRIX.md documenting current state

AC 2.2: Enhance Agent Prompt with Comprehensive Hyperparameter Knowledge (8 points)

Requirements:

[ ] Update agent prompt in GetAgentRecommendationsAsync to include:
- Neural network hyperparameters: hidden layers, activation functions, learning rate schedules
- Tree-based hyperparameters: tree depth, min samples, max features, split criteria
- SVM hyperparameters: kernel type, C, gamma, degree
- Optimizer hyperparameters: batch size, epochs, early stopping criteria
- Regularization hyperparameters: L1/L2 penalties, dropout rates
[ ] Provide agent with model-specific hyperparameter ranges and defaults
[ ] Include hyperparameter interaction guidance (e.g., "deeper trees need more min_samples")

Enhanced Prompt Structure:

var hyperparameterKnowledge = @"
# Hyperparameter Recommendations by Model Type

## Neural Networks (MLPRegressor, MLPClassifier)
- HiddenLayers: Array of layer sizes (e.g., [100, 50] for 2 hidden layers)
- ActivationFunction: ""ReLU"", ""Sigmoid"", ""Tanh""
- LearningRate: 0.001-0.01 (smaller for large datasets)
- MaxEpochs: 100-1000 (use early stopping)
- BatchSize: 32-128 (power of 2, based on dataset size)

## Random Forest (RandomForestClassifier, RandomForestRegressor)
- NumTrees: 100-500 (more trees = better but slower)
- MaxDepth: null (unlimited) or 10-30 (prevent overfitting)
- MinSamplesSplit: 2-10 (higher for noisy data)
- MaxFeatures: ""sqrt"" (classification) or ""1/3"" (regression)

## Support Vector Machines
- KernelType: ""RBF"", ""Linear"", ""Polynomial""
- C: 0.1-100 (regularization strength)
- Gamma: ""scale"" or 0.001-1.0
- Degree: 2-5 (only for polynomial kernel)

## Gradient Boosting
- NumEstimators: 100-1000
- LearningRate: 0.01-0.3 (inversely related to num_estimators)
- MaxDepth: 3-8 (shallow trees for boosting)
- Subsample: 0.5-1.0 (prevent overfitting)

[Include all 80+ model types with specific recommendations]
";

var prompt = $@"
Analyze this dataset and recommend hyperparameters.

{datasetAnalysis}

{hyperparameterKnowledge}

Return JSON with SuggestedHyperparameters as Dictionary<string, object>.
";

Validation:

[ ] Agent provides hyperparameters for at least 20 most common model types
[ ] Hyperparameters are contextually appropriate (e.g., deep trees for small datasets)
[ ] Agent explains reasoning for each hyperparameter choice

AC 2.3: Add Model-Specific Hyperparameter Validation (5 points)

Requirements:

[ ] Enhance HyperparameterApplicator to validate hyperparameter values
[ ] Check ranges (e.g., learning rate must be > 0)
[ ] Check dependencies (e.g., polynomial kernel requires degree parameter)
[ ] Provide helpful error messages for invalid values

Validation Logic:

private void ValidateHyperparameter(string name, object value, Type propertyType)
{
    // Example validations
    if (name.Equals("LearningRate", StringComparison.OrdinalIgnoreCase))
    {
        var lr = Convert.ToDouble(value);
        if (lr <= 0 || lr > 1.0)
            throw new ArgumentOutOfRangeException(
                nameof(value),
                $"LearningRate must be in range (0, 1.0], got {lr}");
    }

    if (name.Equals("MaxDepth", StringComparison.OrdinalIgnoreCase) && value != null)
    {
        var depth = Convert.ToInt32(value);
        if (depth < 1)
            throw new ArgumentOutOfRangeException(
                nameof(value),
                $"MaxDepth must be >= 1, got {depth}");
    }

    // Add more validations for common hyperparameters
}

Phase 3: IConfigurableModel Interface (Optional Future Enhancement)

Goal: Provide a formal interface for models to expose their hyperparameters.

AC 3.1: Design IConfigurableModel<T> Interface (3 points)

Requirements:

[ ] Define interface in src/Interfaces/IConfigurableModel.cs

[ ] Interface signature:

public interface IConfigurableModel<T>
{
    /// <summary>
    /// Gets metadata about configurable hyperparameters
    /// </summary>
    IEnumerable<HyperparameterMetadata> GetHyperparameters();

    /// <summary>
    /// Sets a hyperparameter value
    /// </summary>
    void SetHyperparameter(string name, object value);

    /// <summary>
    /// Gets current value of a hyperparameter
    /// </summary>
    object GetHyperparameter(string name);
}

public class HyperparameterMetadata
{
    public string Name { get; set; } = string.Empty;
    public Type Type { get; set; } = typeof(object);
    public string Description { get; set; } = string.Empty;
    public object? MinValue { get; set; }
    public object? MaxValue { get; set; }
    public object? DefaultValue { get; set; }
    public string[]? AllowedValues { get; set; }
}

Note: This is a future enhancement to enable:

Discovery of available hyperparameters
Validation constraints
Better error messages
Auto-generated UI for hyperparameter tuning

Not included in current scope - would require refactoring 80+ model classes.

Testing Requirements

Unit Tests (src/Tests/UnitTests/Agents/HyperparameterApplicatorTests.cs)

[ ] Test_ApplyHyperparameters_SetsIntProperty: Verify int property is set correctly
[ ] Test_ApplyHyperparameters_SetsDoubleProperty: Verify double property is set correctly
[ ] Test_ApplyHyperparameters_SetsBoolProperty: Verify bool property is set correctly
[ ] Test_ApplyHyperparameters_SetsEnumProperty: Verify enum property set from string
[ ] Test_ApplyHyperparameters_CaseInsensitiveMatching: Property names matched case-insensitively
[ ] Test_ApplyHyperparameters_RespectsOverrideFlag: Existing values not overridden when flag is false
[ ] Test_ApplyHyperparameters_ReadOnlyPropertySkipped: Read-only properties gracefully skipped
[ ] Test_ApplyHyperparameters_InvalidTypeReported: Type mismatches reported as failures
[ ] Test_ApplyHyperparameters_NonExistentPropertyReported: Unknown properties reported as failures
[ ] Test_ApplyHyperparameters_ResultSummary: Result object contains accurate counts and details

Integration Tests (src/Tests/IntegrationTests/Agents/)

[ ] Test_AgentRecommendations_AppliedToRandomForest: Hyperparameters applied to RandomForest model
[ ] Test_AgentRecommendations_AppliedToNeuralNetwork: Hyperparameters applied to MLP model
[ ] Test_AgentRecommendations_NoApplicationWhenDisabled: Respects AutoApplyHyperparameters=false
[ ] Test_AgentRecommendations_PartialApplicationReported: Some succeed, some fail, all reported correctly

Coverage Tests

[ ] Test_HyperparameterCoverage_Top20Models: Verify agent recommends hyperparameters for 20 most common models
[ ] Test_HyperparameterCoverage_ValidValues: Recommended values are within valid ranges

Migration Guide

For Library Users

Before (manual hyperparameter application):

var result = await builder
    .ConfigureAgentAssistance(options => options.EnableModelSelection())
    .BuildAsync(data, labels);

// Manual extraction and application
var recommended = result.AgentRecommendation.SuggestedHyperparameters;
// Now what? How do I apply these to my model?

After (automatic hyperparameter application):

var result = await builder
    .UseModel(new RandomForest<double>())
    .ConfigureAgentAssistance(options => options
        .EnableModelSelection()
        .EnableHyperparameterApplication()) // Auto-apply hyperparameters
    .BuildAsync(data, labels);

// Hyperparameters automatically applied to RandomForest
// Console output shows exactly what was applied

Opt-out (if you want manual control):

var result = await builder
    .UseModel(new RandomForest<double>())
    .ConfigureAgentAssistance(options => options
        .EnableModelSelection()
        .EnableHyperparameterApplication(false)) // Disable auto-application
    .BuildAsync(data, labels);

// Manually inspect and apply
var recommended = result.AgentRecommendation.SuggestedHyperparameters;

Definition of Done

[ ] HyperparameterApplicator<T> class implemented with reflection-based property setting
[ ] HyperparameterApplicationResult class provides detailed reporting
[ ] ApplyAgentRecommendations integrates hyperparameter application
[ ] AgentConfiguration includes AutoApplyHyperparameters option (default: true)
[ ] Agent prompt enhanced with comprehensive hyperparameter knowledge for 20+ model types
[ ] Hyperparameter validation implemented for common parameters
[ ] HYPERPARAMETER_COVERAGE_MATRIX.md documents current coverage
[ ] All unit tests pass with >= 90% code coverage for new components
[ ] Integration tests confirm end-to-end hyperparameter application
[ ] XML documentation complete with beginner-friendly examples
[ ] Console output provides clear feedback on what was applied
[ ] User can opt-in or opt-out of auto-application
[ ] No breaking changes to existing API
[ ] Code review approved

Success Metrics

Code Quality:

Reduce manual hyperparameter configuration burden
Achieve >= 90% test coverage for HyperparameterApplicator
Zero breaking changes to existing model builders

Developer Experience:

Users benefit from AI-recommended hyperparameters automatically
Clear visibility into what hyperparameters were applied
Easy opt-out for users who want manual control

Agent Effectiveness:

Agent provides hyperparameter recommendations for 20+ model types
Recommended values are contextually appropriate for dataset characteristics
80%+ of recommended hyperparameters successfully applied to compatible models

Related Issues

PR #423 - Code review comments that identified ApplyAgentRecommendations limitation
Future: IConfigurableModel interface for formal hyperparameter introspection

Labels

enhancement - Improves existing functionality
agent-assistance - AI agent system improvements
high-priority - Critical for agent system value proposition

Estimated Effort: 31 story points across all phases

Phase 1 (Foundation): 15 points (HyperparameterApplicator + integration)
Phase 2 (Coverage): 16 points (audit + prompt enhancement + validation)
Phase 3 (Future): 3 points (IConfigurableModel design only, not implementation)

Priority: High - Unlocks full value of agent recommendation system

Status: Ready for Implementation

Nov 09 '25 04:11 ooples