chromium-dashboard
chromium-dashboard copied to clipboard
GoogleChrome
Feature popularity stackrank chart looks wrong
https://www.chromestatus.com/metrics/feature/popularity says that the most popular feature implemented in Chromium is CSSSelectorWebkitProgressBar with 1.36%. However, CookieGet, which is used in >85% of the websites doesn't appear in the stackrank.
Weird. These are back for me:
We didn't make any changes to the site overnight :) Maybe there were backend issues with GCP and the datastore.
From reading blink-dev threads, looks like metrics where changed from under us. Reopening until we figure out what's going on.
The % are also over 100% :(
cc: @loonybear -- in case this is related to recent UseCounter changes in Blink.
Yea sorry about that. The % over 100% thing should have been fixed and rolled out soon (M67).
Thanks @loonybear. Guess we'll keep this open and watch for m67 to hit stable. Hopefully this fixes itself.
@ebidel and @loonybear - those percentages are feeding investigations and decisions on Blink intents, can a fix be merged, or can something else be done so that the numbers would be trustworthy?
Most features are still being reported correctly. Big feature usage shift was due to a few bugs that caused the pagevisits being off by 30%.
Over 100% usage were because features being logged before page commits and they could be a) logged multiple times per page or b) counted even though shouldn't be logged at all (features from a non http/https page or a new tab page or etc). But those features are rare, as most features are logged after page commits.
We can certainly merge it earlier but I don't think the percentages will be affecting blink intents process much.
@phistuck I'm not worried about a temporary hiccup in a few metrics. Plus, anything that high wouldn't be considered for a deprecation/removal :)
@ebidel - having a few metrics above 100% looks like the tip of the iceberg and makes me (at least) not trust the rest of the results. Other things might have been blown out of proportion as well, but simply did not make it to over 0.1% more than the usual. This reduces the deprecation/removal opportunities, which sounds like you are against for some reason (I know they can be hard for the few developers that use those features, but they do benefit the platform overall, after all).
Re "having a few metrics above 100% looks like the tip of the iceberg and makes me (at least) not trust the rest of the results": I totally understand that people will tend to distrust the results because of that. However, the reason for the over 100% usage is due to some features measured before load commits, which are very rare cases.
The change from previous to now: Previously: PageVisits were not counted correctly, off by 30% Now: PageVisits is correctly counted (reduced by 30%) There was no change made to the count of any other features, which means the previous stats for the usage (feature count / page visits count) was wrong initially. Now feature usage might be inflated by 30%, but for most cases, feature usage is more accurate despite some of the usage is over 100%.
Re "reduces the deprecation/removal opportunities, which sounds like you are against for some reason", I don't think Eric was against deprecation/removal. It is just normally when a feature has a usage >=0.1%, we consider the impact to be too large to change. In other word, it will break the web. For those features having usage over 100%, even with the fix, I doubt the usage will be reduced to below 0.1%, which means we are not going to change the behaviour of those features anyways. For other features, previously we were using the wrong stats (current usage / 1.3). And now we are using the right stats. But the 1.3 difference will not shift a feature usage from 0.1% to 0.01% or from 1% to 0.1% so it doesn't really affect the intent process much.
I will look into if I can merge the change to an earlier version. But hopefully my comments above can reduce your concern.
Thanks
Thanks for the explanation @loonybear. That's what I was referring to :)
From the sound if it, the data is getting more accurate. In the long-term, that puts us in a better place than before. Short-term, we have a few features reporting higher-than-expected numbers. It's worth noting the opposite has been true in the past. We under reported for a long time e.g. TheJump from last year:
Hopefully issues like this don't happen as often in the future, but we'll always be fine tuning the data.
Getting back here since M67 has been shipped for a few days. I don't think M67 rolled out to everyone yet, but still, considering the stats are calculated ~24hours there should be some change ...
Screenshot of the stack rank today.
The numbers that we're pulling from UMA still are inflated. You can see that the entry on 2018-06-01 is still 142%.
@loonybear Are these numbers starting to stabilizing on internal UMA?
This is concerning. I will test locally and get back to you shortly.
On Mon, Jun 4, 2018 at 2:40 PM, Eric Bidelman [email protected] wrote:
The numbers that we're pulling from UMA still are inflated. You can see that the entry on 2018-06-01 is still 142%.
[image: screen shot 2018-06-04 at 11 29 28 am] https://user-images.githubusercontent.com/238208/40935090-d7f932a0-67eb-11e8-90b3-e8578c8f32c3.png
@loonybear https://github.com/loonybear Are these numbers starting to stabilizing on internal UMA?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleChrome/chromium-dashboard/issues/545#issuecomment-394456226, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab9HP6zy1hS-wsD0xVvWaki0bGbh-YqYks5t5X8HgaJpZM4TDLqK .
Any updates? Today the top-ranking one (SecureContextCheckFailed) is 177% :frowning:
UMA is coming down internally not that the new stable is taking over.
There's a couple of days of delay on our side. We should hopefully start to see better numbers by end of week.
Today the top "CleanScriptElementWithNonce" is ~97% (< 100%), so it's fixed now...?
The old top ranked "SecureContextCheckFailed" is ~25% now, down from 177% 10 days ago, which is a sharp drop.
Shall we ignore the date between Mar. 20 ~ Jun. 15? And it would be nice we have another note like one for 2017-10-26.
Oh great! Looks like it's corrected itself. I'll leave this bug open as a reminder to annotate the graph.
Hi,
Sorry for the late reply. I was on vacation in the past two weeks.
To confirm the fix. Yes, we landed the fix for that in M67. The results on chromestatus.com reflects to the records on "dominant" milestone (which is why the changes are later than when the milestone is rolled out).
And yay. I am glad that everything is finally "correct" again.
There are still minor tweaks I am working on to improve the accuracy of UseCounter further. So please stay tuned.
Cheers,
Luna Lu | Software Engineer | [email protected] | +1 519 513 5751
On Fri, Jun 29, 2018 at 11:53 PM Eric Bidelman [email protected] wrote:
Oh great! Looks like it's corrected itself. I'll leave this bug open as a reminder to annotate the graph.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleChrome/chromium-dashboard/issues/545#issuecomment-401515060, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab9HPxlanbpK50BnTiAaKEFveTaT68xYks5uBvYkgaJpZM4TDLqK .
Emil (eae) and I were looking at graphs today and they still seem to be inflated. Should this have been fixed or did this bug surface again?
Hi, could you please list me some features that you think are inflated?
Thanks
Luna Lu | Software Engineer | [email protected] | +1 519 513 5751
On Tue, Aug 14, 2018 at 12:35 PM Philip Rogers [email protected] wrote:
Emil (eae) and I were looking at graphs today and they still seem to be inflated. Should this have been fixed or did this bug surface again?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleChrome/chromium-dashboard/issues/545#issuecomment-412935459, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab9HP33ZEiCMz1ihmvUfY19RPvNGF2rIks5uQvxEgaJpZM4TDLqK .
A lot of them are over 100%: https://www.chromestatus.com/metrics/css/popularity
For example, display is reported as being on ~150% of pages:
https://www.chromestatus.com/metrics/css/timeline/popularity/4
Yea, this is expected.
The change for the HTML features are merged a lot earlier than the change for the CSS properties.
Commit ed9859e0... https://crrev.com/ed9859e051fd0175204cbb8a0efb2cefc20790de initially landed in 69.0.3457.0
So to wait that change tp be in the dominant version we still need to wait for a little bit.
Sorry for the inconvenience.
Luna Lu | Software Engineer | [email protected] | +1 519 513 5751
On Tue, Aug 14, 2018 at 1:01 PM Philip Rogers [email protected] wrote:
A lot of them are over 100%: https://www.chromestatus.com/metrics/css/popularity
For example, display is reported as being on ~150% of pages: https://www.chromestatus.com/metrics/css/timeline/popularity/4
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleChrome/chromium-dashboard/issues/545#issuecomment-412943836, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab9HP4YEcNi84wUEX74RJNIO5gH6YiH5ks5uQwJ1gaJpZM4TDLqK .
Thanks for the quick response. If I understand correctly, this will be fixed when M69 is promoted to stable (near the end of this month)?
I would assume so. Yes.
Luna Lu | Software Engineer | [email protected] | +1 519 513 5751
On Tue, Aug 14, 2018 at 1:13 PM Philip Rogers [email protected] wrote:
Thanks for the quick response. If I understand correctly, this will be fixed when M69 is promoted to stable (near the end of this month)?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleChrome/chromium-dashboard/issues/545#issuecomment-412947481, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab9HP-e6ewxO3scc2kSRavZ6YlqLb-KIks5uQwVIgaJpZM4TDLqK .