FrameworkBenchmarks icon indicating copy to clipboard operation
FrameworkBenchmarks copied to clipboard

Inconsistent composite score best score computation

Open franz1981 opened this issue 1 year ago • 2 comments

Looking at plaintext, for officefloor: https://www.techempower.com/benchmarks/#hw=ph&test=plaintext&section=data-r22&c=e&o=e&l=xan3i7-cn3

it scores its best at 33,107, but while moving to the Composite Score tab (at https://www.techempower.com/benchmarks/#hw=ph&test=composite&section=data-r22&c=e&o=e&l=xan3i7-cn3), it get 5,648,781 resulting far higher in the overall score than it should be.

it looks like a bug...

franz1981 avatar Dec 05 '23 14:12 franz1981

This is not a bug from what I see. “display_name” and "framework" are separate fields in benchmark_config.json. The composite score just add every best score from every category under the same framework regardless the display names of multiple tests.

That said I don't feel it's the correct behavior. IMO the composite score should either take the display names into account and give multiple scores for one framework(if it's has many display names). Or choose one display name with the best overall score for one framework. This also means one test must implement all bench categories in order to be shown in composite score list.

fakeshadow avatar Dec 05 '23 14:12 fakeshadow

Ah, it's surprising, because I've forcibly added a filter (which is the same for both composite/plaintext), which means, as a user, I expect it to be applied regardless which tab I click ie the best plaintext result of a framework "flavour" which doesn't belong to the filter, should not appear.

franz1981 avatar Dec 05 '23 16:12 franz1981