wpt.fyi icon indicating copy to clipboard operation
wpt.fyi copied to clipboard

Published 20 hours ago • verified publisher icon web-platform-tests

Enable subscribing to new test failures

Open npm1 opened this issue 5 years ago • 6 comments

It would be great to be able to subscribe to new test failures for specific folders of wpt/. The new failures could be caused by a) new tests, b) flaky tests, c) changes in existing tests causing problems. I'm especially interested in looking at results in Chrome, but I'd be ok having a subscription that looks at deltas on all browsers.

There are several use cases for this:

  • Finding flaky tests
  • Noting certain browsers behaving incorrectly (can file bugs on them)
  • Finding discrepancies between Chrome's tests being run on the lab vs test results from WPT.fyi

npm1 avatar Jan 15 '20 22:01 npm1

cc @Hexcles

stephenmcgruer avatar Jan 16 '20 13:01 stephenmcgruer

One problem though is that for the third use case this is not great because WPT.fyi uses Chrome dev so it will not immediately reflect bug fixes.

npm1 avatar Jan 16 '20 15:01 npm1

Is there something special about the results that wpt.fyi offers but Chromium CI does not? For example, flaky tests will be found in Chromium by sheriffs or FindIt, new failures caused by code change will turn the waterfall red, and new failures caused by test change during an import can trigger an automatically filed bug if you opt in. Are you specifically interested in released versions / full Chrome?

Hexcles avatar Jan 16 '20 23:01 Hexcles

I don't know what's special but the results do behave somewhat differently in some cases. A simple example of a difference is that the viewport size is different in Chrome CI vs wpt.fyi so tests relying on the viewport size on Chrome CI will fail on wpt.fyi.

npm1 avatar Jan 17 '20 21:01 npm1

Re last question, I'd be mostly interested in ToT which gives the most 'up to date' results, but I guess Dev is close enough.

npm1 avatar Jan 17 '20 21:01 npm1

@beaufortfrancois requested the same feature in https://github.com/web-platform-tests/wpt.fyi/issues/2972.

I also see a need for this that cannot be met by Chromium CI alone, namely to notify somewhen when a new Chrome-only failure (compared to Firefox+Safari) is introduced. A few years ago we were monitoring this and kept the number of Chrome-only failures around 500, but since then it's increased significantly. Being able to monitor this would be very helpful.

For Interop 2022 (the case in https://github.com/web-platform-tests/wpt.fyi/issues/2972) the requirement is to filter the tests by label.

foolip avatar Sep 19 '22 12:09 foolip