firefox-performance-dashboards icon indicating copy to clipboard operation
firefox-performance-dashboards copied to clipboard

Published 20 hours ago • verified publisher icon mozilla-frontend-infra

Incorrectly combining data for chrome/chromium for certain subtests

Open jandem opened this issue 2 years ago • 10 comments

For example: https://arewefastyet.com/win10/benchmarks/raptor-desktop-jetstream2?numDays=60 https://arewefastyet.com/win10/benchmarks/raptor-desktop-speedometer?numDays=60

The first graph (total benchmark score) is okay and has separate data points for chrome and chromium. For the sub tests, we get a single data point for both. It looks like the data is incorrectly combined somewhere.

jandem avatar Nov 29 '22 11:11 jandem

@beatrice-acasandrei is going to look into this when she's back from PTO next week.

davehunt avatar Nov 30 '22 13:11 davehunt

Ping? It would be great to have this fixed :-)

jandem avatar Jan 17 '23 12:01 jandem

Thanks for the ping. @beatrice-acasandrei any further progress getting to the bottom of this issue?

davehunt avatar Jan 17 '23 14:01 davehunt

The results from chrome and chromium are sharing the same signature hash. Since the logic is relying on the signature hash it is not able to differentiate between chrome and chromium results. Same issue is with the generated link for the Perfherder Graph View. One workaround can be to go to Perfherder's Graph and add data manually for chromium from the 'Add test data' button. I have upcoming PTO and we might take a while before we'll be able to fix this issue.

beatrice-acasandrei avatar Jan 18 '23 16:01 beatrice-acasandrei

It sounds like there may be an underlying bug in Perfherder that's not distinguishing the signatures for subtests for chrome vs chromium. It's also affecting the suite level value, although only when clicking the Perfherder link (both chrome/chromium are displayed but with a shared series). This seems to only affect the benchmark detail view (the overview appears to be fine). This suggests we might be able to resolve this in AWFY but I suspect we need to get to the root cause of the shared signature issue. With PTO affecting the team over the next couple of weeks it's unlikely we're going to have a fix soon. The recommended workaround is to use Perfherder graphs view and add the tests you're interested in.

davehunt avatar Jan 19 '23 18:01 davehunt

@jandem I've created a redash query that takes a platform and suite as parameters and builds a Perfherder graph link for all applications: https://sql.telemetry.mozilla.org/queries/89799. It's not perfect, as it includes Firefox for both autoland and mozilla-central, but it may be useful as a workaround for this issue?

davehunt avatar Jan 19 '23 20:01 davehunt

@davehunt thanks for doing that! I'll give it a try :)

jandem avatar Jan 20 '23 07:01 jandem

I've improved the query to only show autoland for Firefox and only mozilla-central for non-Firefox.

davehunt avatar Jan 20 '23 12:01 davehunt

@jandem Hi! Does the query provided by Dave fulfill your needs for now? This issue is more complex than we initially anticipated and might take a while until it's fixed.

beatrice-acasandrei avatar Feb 06 '23 12:02 beatrice-acasandrei

@beatrice-acasandrei I think that works for now. Maybe we can put a banner on the relevant pages to warn people about this bug, because the data is kind of misleading atm?

jandem avatar Feb 06 '23 15:02 jandem