aqa-test-tools icon indicating copy to clipboard operation
aqa-test-tools copied to clipboard

Perf Compare (TRSS) regression percentage calculation for startup and footprint benchmarks

Open eramos2 opened this issue 4 years ago • 1 comments

We see that currently on Perf Compare (TRSS) the regression percentage for SUFP is being calculated by dividing the baseline result by the test result (above 100% is an improvement). In the WAS Performance team we have been doing it the other way around, test result / baseline result (below 100% is an improvement). We also add a +/- 3% threshold to account for test variability.

const curDiff = curHigherBetter ? curTestScore / curBaselineScore : curBaselineScore / curTestScore;
// Row colours based on improvement or regression
if (curDiff > 1) {
    curColor = 'improvement';
} else if (curDiff < 1) {
    curColor = 'regression';
} else {
    curColor = 'nochange';
}
const curDiff = curTestScore / curBaselineScore;
// WAS Performance Threshold
const regressionThreshold = 0.03;
// Row colours based on improvement or regression
if ( (curHigherBetter && curDiff > 1 + regressionThreshold) || (!curHigherBetter && curDiff < 1 - regressionThreshold) ) {
    curColor = 'improvement';
} else if ( (curHigherBetter && curDiff < 1 - regressionThreshold) || (!curHigherBetter && curDiff > 1 + regressionThreshold) ) {
    curColor = 'regression';
} else {
    curColor = 'nochange';
}

eramos2 avatar Jul 27 '20 17:07 eramos2

Thanks @eramos2 ! I've labeled with discussion tag and will others to weigh in before we decide how best to proceed. Cheers!

smlambert avatar Jul 27 '20 18:07 smlambert