ART
ART copied to clipboard
Improve Scenario Generation Using MCP Server
This pull request improves the scenario/test case generation process by using a local mcp (stdio) format server to generate the response needed for scenarios.
Included in the examples directory are two modified files that use local llm's with local mcp servers.
python3 generate_from_http_mcp.py
======================================================================
π MCP Scenario Generator
======================================================================
π Server: http://mcp-code-coverage:8000/mcp
π€ LLM: gpt-oss
π― Scenarios: 10
π Connecting to MCP server at http://mcp-code-coverage:8000/mcp...
β Connected successfully
β Found 4 tools and 1 resources
======================================================================
π Discovered Capabilities
======================================================================
π§ Tools (4):
1. submit_testcase
Submit a test case for coverage analysis.
Args...
2. get_function_coverage
Get detailed line-by-line coverage for a specific funct...
3. get_file_coverage
Get coverage report for a specific source file.
...
4. get_coverage_summary
Get overall coverage summary across all files.
...
π Resources (1):
1. get_coverage_summary
Real-time coverage summary updated after each test case...
======================================================================
π― Generating Scenarios
======================================================================
[18:21:18] OK Using model: gpt-oss
[18:21:18] INFO Available: 4 tool(s), 1 resource(s).
[18:21:18] STEP Preparing prompt for scenario generation
[18:21:18] STEP Calling model: gpt-oss
[18:21:25] OK Model responded in 7.66s.
[18:21:25] INFO Raw content length: 3452 chars.
[18:21:25] STEP Parsing model output
[18:21:25] STEP Connecting to MCP server for JSON formatting
[18:21:25] OK Using MCP tool: format_scenario
[18:21:25] INFO Formatted 5/10 scenarios
[18:21:25] INFO Formatted 10/10 scenarios
[18:21:26] OK Formatted 10 scenarios via MCP server.
[18:21:26] INFO Difficulty distribution:
1/5: 1 β
2/5: 1 β
3/5: 3 βββ
4/5: 3 βββ
5/5: 2 ββ
1. Submit a single HTML test case that intentionally omits the `<title>` tag and use `submit_testcase` to trigger coverage⦠(difficulty 1/5)
2. Run a JavaScript test case (`test_type="js"`) designed to exercise a rarely used branch in the `calculateDiscount` funct⦠(difficulty 2/5)
3. Submit a PNG rendering test (`test_type="png"`) that draws a complex UI component. After submission, use `get_file_cover⦠(difficulty 3/5)
4. Evaluate overall health of the codebase by invoking `get_coverage_summary`. Parse the returned statistics to produce a c⦠(difficulty 3/5)
5. Create a raw data test (`test_type="raw"`) that feeds malformed JSON to the `parseConfig` function. After running the te⦠(difficulty 4/5)
[18:21:26] OK Generated 10 scenarios in 7.73s total.
======================================================================
πΎ Saving Scenarios
======================================================================
β Saved to: scenarios.json
β File size: 4,290 bytes
======================================================================
π Summary
======================================================================
π Statistics:
Total scenarios: 10
Average difficulty: 3.4/5
Average task length: 306 characters
π Difficulty Distribution:
1/5: 1 ( 10.0%) ββββββββββ
2/5: 1 ( 10.0%) ββββββββββ
3/5: 3 ( 30.0%) ββββββββββββββββββββββββββββββ
4/5: 3 ( 30.0%) ββββββββββββββββββββββββββββββ
5/5: 2 ( 20.0%) ββββββββββββββββββββ
β
Complete!
root@art:/workspace/code-coverage-agent# cat scenarios.json
{
"metadata": {
"generated_at": "2025-11-01T18:21:26.002254",
"mcp_server_url": "http://mcp-code-coverage:8000/mcp",
"llm_model": "gpt-oss",
"num_tools": 4,
"num_resources": 1,
"tool_names": [
"submit_testcase",
"get_function_coverage",
"get_file_coverage",
"get_coverage_summary"
],
"resource_names": [
"get_coverage_summary"
],
"summary": {
"total_scenarios": 10,
"difficulty_distribution": {
"1": 1,
"2": 1,
"3": 3,
"4": 3,
"5": 2
},
"avg_difficulty": 3.4,
"avg_task_length": 305.5
}
},
"scenarios": [
{
"task": "Submit a single HTML test case that intentionally omits the `<title>` tag and use `submit_testcase` to trigger coverage of the HTML validator. After execution, retrieve the real\u2011time summary from `coverage://summary` to confirm that the tidy warning count increased.",
"difficulty": 1
},
{
"task": "Run a JavaScript test case (`test_type=\"js\"`) designed to exercise a rarely used branch in the `calculateDiscount` function. Then call `get_function_coverage` for `calculateDiscount` to verify that the new branch was hit and see its line\u2011by\u2011line coverage diff.",
"difficulty": 2
},