ART icon indicating copy to clipboard operation
ART copied to clipboard

Improve Scenario Generation Using MCP Server

Open cielonet opened this issue 4 months ago β€’ 1 comments

This pull request improves the scenario/test case generation process by using a local mcp (stdio) format server to generate the response needed for scenarios.

Included in the examples directory are two modified files that use local llm's with local mcp servers.

cielonet avatar Nov 01 '25 18:11 cielonet

python3 generate_from_http_mcp.py 
======================================================================
πŸš€ MCP Scenario Generator
======================================================================

πŸ“ Server: http://mcp-code-coverage:8000/mcp
πŸ€– LLM: gpt-oss
🎯 Scenarios: 10

πŸ” Connecting to MCP server at http://mcp-code-coverage:8000/mcp...
βœ“ Connected successfully
βœ“ Found 4 tools and 1 resources

======================================================================
πŸ“‹ Discovered Capabilities
======================================================================

πŸ”§ Tools (4):
   1. submit_testcase
      
    Submit a test case for coverage analysis.
    
    Args...
   2. get_function_coverage
      
    Get detailed line-by-line coverage for a specific funct...
   3. get_file_coverage
      
    Get coverage report for a specific source file.
    
  ...
   4. get_coverage_summary
      
    Get overall coverage summary across all files.
    
   ...

πŸ“š Resources (1):
   1. get_coverage_summary
      
    Real-time coverage summary updated after each test case...

======================================================================
🎯 Generating Scenarios
======================================================================

[18:21:18] OK    Using model: gpt-oss
[18:21:18] INFO  Available: 4 tool(s), 1 resource(s).
[18:21:18] STEP  Preparing prompt for scenario generation
[18:21:18] STEP  Calling model: gpt-oss
[18:21:25] OK    Model responded in 7.66s.
[18:21:25] INFO  Raw content length: 3452 chars.
[18:21:25] STEP  Parsing model output
[18:21:25] STEP  Connecting to MCP server for JSON formatting
[18:21:25] OK    Using MCP tool: format_scenario
[18:21:25] INFO  Formatted 5/10 scenarios
[18:21:25] INFO  Formatted 10/10 scenarios
[18:21:26] OK    Formatted 10 scenarios via MCP server.
[18:21:26] INFO  Difficulty distribution:
   1/5:   1  β–ˆ
   2/5:   1  β–ˆ
   3/5:   3  β–ˆβ–ˆβ–ˆ
   4/5:   3  β–ˆβ–ˆβ–ˆ
   5/5:   2  β–ˆβ–ˆ
   1. Submit a single HTML test case that intentionally omits the `<title>` tag and use `submit_testcase` to trigger coverage…  (difficulty 1/5)
   2. Run a JavaScript test case (`test_type="js"`) designed to exercise a rarely used branch in the `calculateDiscount` funct…  (difficulty 2/5)
   3. Submit a PNG rendering test (`test_type="png"`) that draws a complex UI component. After submission, use `get_file_cover…  (difficulty 3/5)
   4. Evaluate overall health of the codebase by invoking `get_coverage_summary`. Parse the returned statistics to produce a c…  (difficulty 3/5)
   5. Create a raw data test (`test_type="raw"`) that feeds malformed JSON to the `parseConfig` function. After running the te…  (difficulty 4/5)
[18:21:26] OK    Generated 10 scenarios in 7.73s total.

======================================================================
πŸ’Ύ Saving Scenarios
======================================================================
βœ“ Saved to: scenarios.json
βœ“ File size: 4,290 bytes

======================================================================
πŸ“Š Summary
======================================================================

πŸ“ˆ Statistics:
  Total scenarios: 10
  Average difficulty: 3.4/5
  Average task length: 306 characters

πŸ“Š Difficulty Distribution:
  1/5:   1 ( 10.0%)  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ
  2/5:   1 ( 10.0%)  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ
  3/5:   3 ( 30.0%)  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ
  4/5:   3 ( 30.0%)  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ
  5/5:   2 ( 20.0%)  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ

βœ… Complete!
root@art:/workspace/code-coverage-agent# cat scenarios.json 
{
  "metadata": {
    "generated_at": "2025-11-01T18:21:26.002254",
    "mcp_server_url": "http://mcp-code-coverage:8000/mcp",
    "llm_model": "gpt-oss",
    "num_tools": 4,
    "num_resources": 1,
    "tool_names": [
      "submit_testcase",
      "get_function_coverage",
      "get_file_coverage",
      "get_coverage_summary"
    ],
    "resource_names": [
      "get_coverage_summary"
    ],
    "summary": {
      "total_scenarios": 10,
      "difficulty_distribution": {
        "1": 1,
        "2": 1,
        "3": 3,
        "4": 3,
        "5": 2
      },
      "avg_difficulty": 3.4,
      "avg_task_length": 305.5
    }
  },
  "scenarios": [
    {
      "task": "Submit a single HTML test case that intentionally omits the `<title>` tag and use `submit_testcase` to trigger coverage of the HTML validator. After execution, retrieve the real\u2011time summary from `coverage://summary` to confirm that the tidy warning count increased.",
      "difficulty": 1
    },
    {
      "task": "Run a JavaScript test case (`test_type=\"js\"`) designed to exercise a rarely used branch in the `calculateDiscount` function. Then call `get_function_coverage` for `calculateDiscount` to verify that the new branch was hit and see its line\u2011by\u2011line coverage diff.",
      "difficulty": 2
    },

cielonet avatar Nov 01 '25 18:11 cielonet