FrameworkBenchmarks
FrameworkBenchmarks copied to clipboard
Inconsistent composite score best score computation
Looking at plaintext
, for officefloor
: https://www.techempower.com/benchmarks/#hw=ph&test=plaintext§ion=data-r22&c=e&o=e&l=xan3i7-cn3
it scores its best at 33,107
, but while moving to the Composite Score
tab (at https://www.techempower.com/benchmarks/#hw=ph&test=composite§ion=data-r22&c=e&o=e&l=xan3i7-cn3), it get 5,648,781
resulting far higher in the overall score than it should be.
it looks like a bug...
This is not a bug from what I see. “display_name” and "framework" are separate fields in benchmark_config.json
. The composite score just add every best score from every category under the same framework regardless the display names of multiple tests.
That said I don't feel it's the correct behavior. IMO the composite score should either take the display names into account and give multiple scores for one framework(if it's has many display names). Or choose one display name with the best overall score for one framework. This also means one test must implement all bench categories in order to be shown in composite score list.
Ah, it's surprising, because I've forcibly added a filter (which is the same for both composite/plaintext), which means, as a user, I expect it to be applied regardless which tab I click ie the best plaintext result of a framework "flavour" which doesn't belong to the filter, should not appear.