web-platform-tests
Score beta runs
@andreban asked if there's a way to know what the score of the next stable Chrome release is going to be before it happens. The answer is no, because we only score stable runs and experimental runs. However, if we scored beta runs, we would have a very good idea of what the changes are going to be ahead of time.
@jgraham would this be useful for Firefox?
@DanielRyanSmith what do you think about computing these scores? I don't think we need to show them on the dashboard.
It looks like runs with the beta label are only available for Chrome and Firefox, and are run once a week (see https://wpt.fyi/runs?label=master&label=beta&max-count=10&product=chrome&product=firefox&product=safari)
These questions are the first to pop in my head:
- Would weekly runs be sufficient enough to gauge upcoming stable results? Or will more beta runs be needed?
- Would it be worth the computing cost to get more beta results if weekly runs are not enough?
- Are interop scores important to see in this case? Because we would not be able to calculate changes in interop scores without a proper comparison for Safari.
I don't know anything about the value of this information 😅 so these are just my questions on possible implementation
On the question around the value of this information, we are looking to update developers on Chrome's interoperability improvements on the New in Chrome blog posts and videos, which are produced 1 or 2 weeks before the stable release. Since currently the scores are only available after the stable release, it impossible to include in a timely manner in the communication.
