aqa-tests
aqa-tests copied to clipboard
Generic logic to get get and store benchmark metric
Extend based on https://github.com/adoptium/aqa-tests/pull/6339/, make checkDacapoH2Metric() and archiveDacapoH2Metric() generic.
On second thought, it may be better to have a separate script (i.e., Python - benchmarkMetric,py) to handle this.
- In
benchmarkMetric.py, it contains a generic functioncheckMetric().checkMetric()takes job console output and readsbenchmarkMetric.jsonand runs each regex, and stores into one JSON output.
benchmarkMetric.json example:
[
{
"Benchmark": "dacapo-eclipse",
"metrics":
{
"dacapo-eclipse" : [
{"regex": "DaCapo.*eclipse\\sPASSED\\sin\\s(\\d*\\.?\\d*)\\smsec"},
{"higherbetter": true},
{"units": "msec"}
]
}
},
{
"Benchmark": "dacapo-h2",
"metrics":
{
"dacapo-h2" : [
{"regex": "DaCapo.*h2\\sPASSED\\sin\\s(\\d*\\.?\\d*)\\smsec"},
{"higherbetter": true},
{"units": "msec"}
]
}
}
]
Note: Backslashes must be double-escaped (\\) in JSON strings.
Output: JSON file as specified in https://github.com/adoptium/aqa-tests/issues/6327. For example:
[
{
"Benchmark": "dacapo-eclipse",
"metrics":
{
"dacapo-eclipse" : [
{"value": "123456"},
{"higherbetter": true},
{"units": "msec"}
]
}
},
{
"Benchmark": "dacapo-h2",
"metrics":
{
"dacapo-h2" : [
{"value": "4689"},
{"higherbetter": true},
{"units": "msec"}
]
}
}
]
related: https://github.com/adoptium/aqa-tests/issues/6327
top level issue: https://github.com/adoptium/aqa-tests/issues/6268
@smlambert @LongyuZhang if I missed anything, please feel free to update.