aqa-test-tools
aqa-test-tools copied to clipboard
Perf Compare (TRSS) regression percentage calculation for startup and footprint benchmarks
We see that currently on Perf Compare (TRSS) the regression percentage for SUFP is being calculated by dividing the baseline result by the test result (above 100% is an improvement). In the WAS Performance team we have been doing it the other way around, test result / baseline result (below 100% is an improvement). We also add a +/- 3% threshold to account for test variability.
const curDiff = curHigherBetter ? curTestScore / curBaselineScore : curBaselineScore / curTestScore;
// Row colours based on improvement or regression
if (curDiff > 1) {
curColor = 'improvement';
} else if (curDiff < 1) {
curColor = 'regression';
} else {
curColor = 'nochange';
}
const curDiff = curTestScore / curBaselineScore;
// WAS Performance Threshold
const regressionThreshold = 0.03;
// Row colours based on improvement or regression
if ( (curHigherBetter && curDiff > 1 + regressionThreshold) || (!curHigherBetter && curDiff < 1 - regressionThreshold) ) {
curColor = 'improvement';
} else if ( (curHigherBetter && curDiff < 1 - regressionThreshold) || (!curHigherBetter && curDiff > 1 + regressionThreshold) ) {
curColor = 'regression';
} else {
curColor = 'nochange';
}
Thanks @eramos2 ! I've labeled with discussion tag and will others to weigh in before we decide how best to proceed. Cheers!