aqa-tests icon indicating copy to clipboard operation
aqa-tests copied to clipboard

Generic logic to get get and store benchmark metric

Open llxia opened this issue 4 months ago • 1 comments

Extend based on https://github.com/adoptium/aqa-tests/pull/6339/, make checkDacapoH2Metric() and archiveDacapoH2Metric() generic.

On second thought, it may be better to have a separate script (i.e., Python - benchmarkMetric,py) to handle this.

  • In benchmarkMetric.py, it contains a generic function checkMetric(). checkMetric() takes job console output and reads benchmarkMetric.json and runs each regex, and stores into one JSON output.

benchmarkMetric.json example:

[
    {
        "Benchmark": "dacapo-eclipse",
        "metrics": 
        {
          "dacapo-eclipse" : [
            {"regex": "DaCapo.*eclipse\\sPASSED\\sin\\s(\\d*\\.?\\d*)\\smsec"},
            {"higherbetter": true},
            {"units": "msec"}
        ]
      }
    },
{
        "Benchmark": "dacapo-h2",
        "metrics": 
        {
          "dacapo-h2" : [
            {"regex": "DaCapo.*h2\\sPASSED\\sin\\s(\\d*\\.?\\d*)\\smsec"},
            {"higherbetter": true},
            {"units": "msec"}
        ]
      }
    }
]

Note: Backslashes must be double-escaped (\\) in JSON strings.

Output: JSON file as specified in https://github.com/adoptium/aqa-tests/issues/6327. For example:

[
    {
        "Benchmark": "dacapo-eclipse",
        "metrics": 
        {
          "dacapo-eclipse" : [ 
            {"value": "123456"},
            {"higherbetter": true},
            {"units": "msec"}
        ]
      }
    },
{
        "Benchmark": "dacapo-h2",
        "metrics": 
        {
          "dacapo-h2" : [
            {"value": "4689"},
            {"higherbetter": true},
            {"units": "msec"}
        ]
      }
    }
]

related: https://github.com/adoptium/aqa-tests/issues/6327

top level issue: https://github.com/adoptium/aqa-tests/issues/6268

@smlambert @LongyuZhang if I missed anything, please feel free to update.

llxia avatar Jun 23 '25 13:06 llxia