browser-compat-data
browser-compat-data copied to clipboard
mdn
Increase quality of compat data
The MDN team is employed by Mozilla and has yearly goals. I'm opening this issue to inform you about this year's goal and to give status updates throughout the year.
In 2018, the team's goal for mdn/browser-compat-data has been to finish migrating compat data from the legacy wiki tables into this repository and offering the data to 3rd party tools. ~This goal is now almost complete (see this gist for remaining pages using the legacy tables and the list of tools using the data.) Work is underway to fully finish this.~ This goal is now complete.
In 2019 and going forward, the goal will be to increase the quality of the compat data found in this repository. Compat data quality can be measured in various ways (I will write about this in the wiki soon), but we're going to focus on our efforts on getting 100% real values for Firefox, IE, Edge, Chrome, Safari, mobile Safari, mobile Chrome for all web platform features (i.e. WebExtensions are excluded). Real values here means to replace all "true", "null" or no data values with either a version number or "false".
We have finished the following categories:
- [x] api
- [X] css
- [ ] html
- [ ] http
- [ ] svg
- [X] javascript
- [X] mathml
- [X] webdriver
- [ ] webextensions
- [x] ~~xpath~~ (removed in #9830)
- [x] ~~xslt~~ (removed in #9830)
Beginning of the year overall baseline data (based on version 0.0.66, released on 7. Feb. 2019) for all 8 tracked browsers for web platform features: :
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 52.36% | 23.77% | 23.87% |
| chrome | 64.55% | 24.47% | 10.98% |
| chrome android | 48.28% | 32.89% | 18.83% |
| edge | 39.38% | 29.75% | 30.89% |
| firefox | 73.08% | 13.63% | 13.28% |
| ie | 59.35% | 15.14% | 25.52% |
| safari | 48.17% | 19.28% | 32.55% |
| safari ios | 39.92% | 20.55% | 39.54% |
| webview android | 46.17% | 34.44% | 19.39% |
Going forward, I will update this issue with every release of BCD, so that we can track where we are.
If you have questions about how this goal was decided, or if you have feedback, please reach out to me or other MDN team members.
Status as of version 0.0.69 (released on 28th February 2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 54.49% (+2.13) | 22.21% (-1.56%) | 23.31% (-0.56%) |
| chrome | 65.20% | 24.05% | 10.75% |
| chrome android | 49.27% | 32.31% | 18.41% |
| edge | 53.07% | 20.06% | 26.88% |
| firefox | 73.44% | 12.97% | 13.59% |
| ie | 59.41% | 14.88% | 25.72% |
| safari | 48.44% | 18.92% | 32.64% |
| safari ios | 40.19% | 20.29% | 39.54% |
| webview android | 46.89% | 34.19% | 18.92% |
I've highlighted Edge here to show the most significant changes to the prior statistics. Updating Edge data was mostly driven by @foolip. Thanks for your work, Philip!
@Elchi3, this is excellent! Thank you for getting started on this effort and sharing the baseline publicly. Would you mind sharing the code you're using to generate the stats, too? Just a gist would be fine—I think generating quality stats could be baked into the npm run release-notes script and save you some time at releases
Thanks Daniel! It makes total sense to have this as a script! I've opened https://github.com/mdn/browser-compat-data/pull/3557 as a first stab at this. When you commented here, I actually realized that we should review the data gathering of this as we surely want these statistics to be correct :smile:
Previously, there was a script that created a csv which then was used to make charts in a sheet, so this is inspired from that prior art (see https://github.com/atopal/browser-compat-data-dashboard).
Thanks for the stats @Elchi3!
I wonder how you'd like to deal with increasing the API coverage of BCD? When adding new entries entirely, it is arguably increasing the the usefulness of the data, but if data for all browsers isn't added up front it will regress these metrics slightly. One option for dealing with this is using absolute values, and aiming for what 100% would mean today.
Thanks @foolip! You're right, by adding more data, this metric can regress.
I think I'd want to be ambitious and see if we can reach this goal anyway and see if can maybe disallow adding new "true" and "nulls" once we've finished parts of this. So, e.g. "all Edge data" or all "Edge CSS data" has 100% real values and then the linter would disallow adding new "true" and "null" values for these parts and so we can't regress anymore. This would force new data contributors to provide real values and I don't know how much of a burden that would be on them. It would be another option, however.
Also tagging @atopal to comment on the option of tracking absolute values of today.
That is an interesting problem. The key result is an indicator for substantial progress against what we really care about: making the compatibility data more useful for individuals and tooling vendors. So, I think we want to reflect current reality as part of that key result and that means using the current data set at any given time. As an added benefit, I think that it will motivate us to figure out how to keep the quality up and not regress over time.
That said, we might want to set a goal that allows for some minor fluctuations. 100% is just a very nice and very round number 😃
If you treat 0.7 as the expected average score of KRs (do you?) then to either 70% or ~85% real values, depending on if you treat 0% or 51.11%, would be success.
Yes, .7 is the expected average score. I think it's fair to take 51% as the baseline. We are shooting for 100%, but a minimum of 85% would already be a significant improvement.
I've updated the above stats for 0.0.66 and 0.0.69 using the latest version of the the script and here is the latest numbers for yesterday's release.
Status as of version 0.0.70 (released on 07/03/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 55.02% (+0.53%) | 22.08% (-0.13%) | 22.90% (-0.41%) |
| chrome | 65.68% | 23.92% | 10.40% |
| chrome android | 49.91% | 31.99% | 18.09% |
| edge | 53.46% | 19.96% | 26.59% |
| firefox | 73.60% | 12.90% | 13.50% |
| ie | 60.19% | 14.79% | 25.02% |
| safari | 49.03% | 18.88% | 32.08% |
| safari ios | 40.74% | 20.34% | 38.93% |
| webview android | 47.51% | 33.90% | 18.59% |
Florian,
Is your script publicly available? I'd like to run it against releases for the last year to find out how far we've come and how fast.
Joe
On Fri, Mar 8, 2019 at 3:15 AM Florian Scholz [email protected] wrote:
I've updated the above stats for 0.0.66 and 0.0.69 using the latest version of the the script and here is the latest numbers for yesterday's release.
Status as of version 0.0.70 (released on 07/03/2019) for web platform features: browser real values true values null values All 8 browsers 55.02% 22.08% 22.90% chrome 65.68% 23.92% 10.40% chrome android 49.91% 31.99% 18.09% edge 53.46% 19.96% 26.59% firefox 73.60% 12.90% 13.50% ie 60.19% 14.79% 25.02% safari 49.03% 18.88% 32.08% safari ios 40.74% 20.34% 38.93% webview android 47.51% 33.90% 18.59%
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mdn/browser-compat-data/issues/3555#issuecomment-470894309, or mute the thread https://github.com/notifications/unsubscribe-auth/AH0viwg2Jwx1SxwKgyhCaWq8pNNXCZOvks5vUkY5gaJpZM4bZAak .
Yes, it's available here: https://github.com/mdn/browser-compat-data/pull/3557
Nice work! Joe Medley | Technical Writer, Chrome DevRel | [email protected] | 816-678-7195 If an API's not documented it doesn't exist.
On Thu, Mar 21, 2019 at 12:30 AM Vinyl Darkscratch [email protected] wrote:
I went ahead and compiled a list of all the features that are still listed as null for Firefox, Safari, Chrome, IE, and Edge (all the desktop versions), putting them in a public gist which can be found here https://gist.github.com/vinyldarkscratch/51e8e607a3fcc4a7d186e189a9b71a83 .
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mdn/browser-compat-data/issues/3555#issuecomment-475134182, or mute the thread https://github.com/notifications/unsubscribe-auth/AH0vi56iSXpm1TBloLYNu6_csGUHG9Vjks5vYzUngaJpZM4bZAak .
[Apologies to everyone for the lack of reviews from my side last week and this week. I was on vacation and then at a workshop presenting about this project :)]
Status as of version 0.0.72 (released on 21/03/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 55.29% (+0.27%) | 22.26% (+0.18%) | 22.46% (-0.44%) |
| chrome | 65.99% | 23.90% | 10.11% |
| chrome android | 50.51% | 31.92% | 17.57% |
| edge | 53.27% | 20.28% | 26.46% |
| firefox | 73.65% | 13.30% | 13.06% |
| ie | 60.60% | 14.81% | 24.59% |
| safari | 49.40% | 19.10% | 31.49% |
| safari ios | 41.11% | 20.48% | 38.43% |
| webview android | 47.77% | 34.27% | 17.96% |
Status as of version 0.0.73 (released on 28/03/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 55.87% (+0.58%) | 22.49% (+0.23%) | 21.64% (-0.84%) |
| chrome | 66.15% | 24.43% | 9.42% |
| chrome android | 51.11% | 32.40% | 16.49% |
| edge | 55.70% | 20.01% | 24.30% |
| firefox | 73.89% | 13.52% | 12.59% |
| ie | 60.66% | 14.81% | 24.53% |
| safari | 49.77% | 19.16% | 31.07% |
| safari ios | 41.59% | 20.64% | 37.79% |
| webview android | 48.09% | 34.95% | 16.96% |
Status as of version 0.0.74 (released on 04/04/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 55.92% (+0.05%) | 24.45% (+1.96%) | 19.64% (-2%) |
| chrome | 66.37% | 24.31% | 9.32% |
| chrome android | 51.50% | 32.26% | 16.24% |
| edge | 55.59% | 19.92% | 24.50% |
| firefox | 73.71% | 13.61% | 12.68% |
| ie | 60.52% | 14.75% | 24.74% |
| safari | 49.77% | 25.91% | 24.33% |
| safari ios | 41.55% | 30.00% | 28.49% |
| webview android | 48.37% | 34.81% | 16.82% |
In this release we were able to move 2% of the values from null to true and I want to thank @foolip and @vinyldarkscratch for their work specifically.
Someone should write a script that can generate a line graph of these percentages over time :) (bonus points if you go back all the way to ~0.50 as well)
@vinyldarkscratch did you generate that gist with a script, and if so could you share it? :)
I did, yeah -- I wrote linter/test-version-bool.js initially as a test against version_added: true, hence the name, but then adapted it to allow for any value. I basically just take the output and run some Sublime Text magic to extract the specific features afterwards (not the best, but I tend to unintentionally do things manually :P).
Status as of version 0.0.75 (released on 11/04/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 55.96% (+0.04%) | 25.85% (+1.40%) | 18.19% (-1.45%) |
| chrome | 66.35% | 26.36% | 7.29% |
| chrome android | 51.54% | 34.96% | 13.50% |
| edge | 55.56% | 20.56% | 23.89% |
| firefox | 73.70% | 16.43% | 9.87% |
| ie | 60.58% | 14.69% | 24.73% |
| safari | 49.87% | 25.87% | 24.28% |
| safari ios | 41.64% | 29.95% | 28.44% |
| webview android | 48.42% | 38.02% | 13.56% |
Status as of version 0.0.76 (released on 18/04/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 56.18% (+0.22%) | 25.69% (-0.16%) | 18.14% (-0.05%) |
| chrome | 66.56% | 26.21% | 7.23% |
| chrome android | 51.82% | 34.83% | 13.35% |
| edge | 55.85% | 20.60% | 23.56% |
| firefox | 74.44% | 15.58% | 9.98% |
| ie | 60.64% | 14.58% | 24.78% |
| safari | 49.89% | 25.80% | 24.31% |
| safari ios | 41.71% | 29.88% | 28.44% |
| webview android | 48.55% | 38.01% | 13.44% |
I started tackling a little something to show the progress of real values overtime (since not just 0.0.50, but all the way back to 0.0.8 -- mind if I snatch those bonus points, @connorshea? 😛), and have been able to get a bar graph generated. The graphs can be found on my website: https://www.queengoob.org/bcd_stats
I plan to automate the process, so that once a new release hits, this data will automatically update. Currently, a manual run of the program is required (and an update of BCD), but once it's run, it records that BCD version and retains it in JSON format.
Status as of version 0.0.77 (released on 25/04/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 57.02% (+0.84%) | 25.68% (-0.01%) | 17.30% (-0.84%) |
| chrome | 66.78% | 26.03% | 7.19% |
| chrome android | 54.56% | 35.11% | 10.33% |
| edge | 56.40% | 20.40% | 23.20% |
| firefox | 74.46% | 15.55% | 9.99% |
| ie | 61.78% | 14.35% | 23.87% |
| safari | 50.29% | 25.56% | 24.16% |
| safari ios | 41.82% | 30.03% | 28.18% |
| webview android | 50.07% | 38.42% | 11.50% |
Status as of version 0.0.78 (released on 02/05/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 57.83% (+0.81%) | 25.73% (+0.05%) | 16.45% (-0.85%) |
| chrome | 67.29% | 25.93% | 6.77% |
| chrome android | 55.44% | 35.25% | 9.31% |
| edge | 57.72% | 19.88% | 22.41% |
| firefox | 74.59% | 15.54% | 9.86% |
| ie | 64.15% | 14.36% | 21.49% |
| safari | 50.44% | 25.48% | 24.09% |
| safari ios | 41.86% | 30.07% | 28.10% |
| webview android | 51.12% | 39.31% | 9.57% |
Status as of version 0.0.79 (released on 09/05/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 58.09% (+0.26%) | 25.63% (-0.10%) | 16.30% (-0.15%) |
| chrome | 67.42% | 25.79% | 6.79% |
| chrome android | 55.58% | 35.10% | 9.32% |
| edge | 57.89% | 19.86% | 22.26% |
| firefox | 74.76% | 15.41% | 9.84% |
| ie | 65.29% | 14.34% | 20.37% |
| safari | 50.57% | 25.32% | 24.12% |
| safari ios | 42.01% | 29.93% | 28.08% |
| webview android | 51.17% | 39.25% | 9.58% |
Status as of version 0.0.80 (released on 16/05/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 58.67% (+0.59%) | 25.34% (-0.29%) | 16.00% (-0.30%) |
| chrome | 68.20% | 25.42% | 6.38% |
| chrome android | 56.43% | 34.67% | 8.90% |
| edge | 58.18% | 19.76% | 22.08% |
| firefox | 74.93% | 15.36% | 9.71% |
| ie | 65.54% | 14.27% | 20.19% |
| safari | 51.16% | 25.00% | 23.85% |
| safari ios | 42.74% | 29.58% | 27.71% |
| webview android | 52.19% | 38.66% | 9.15% |
Status as of version 0.0.81 (released on 23/05/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 58.88% | 25.20% | 15.92% |
| chrome | 68.30% | 25.34% | 6.36% |
| chrome android | 56.56% | 34.57% | 8.87% |
| edge | 58.67% | 19.59% | 21.75% |
| firefox | 75.44% | 14.91% | 9.65% |
| ie | 65.62% | 14.23% | 20.15% |
| safari | 51.29% | 24.92% | 23.80% |
| safari ios | 42.88% | 29.51% | 27.64% |
| webview android | 52.32% | 38.56% | 9.12% |
Status as of version 0.0.82 (released on 30/05/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 58.89% | 25.12% | 15.99% |
| chrome | 68.15% | 25.33% | 6.52% |
| chrome android | 56.67% | 34.30% | 9.03% |
| edge | 59.08% | 19.43% | 21.49% |
| firefox | 75.48% | 14.88% | 9.64% |
| ie | 65.51% | 14.21% | 20.28% |
| safari | 51.19% | 24.88% | 23.94% |
| safari ios | 42.82% | 29.46% | 27.75% |
| webview android | 52.24% | 38.49% | 9.27% |
Status as of version 0.0.83 (released on 06/06/2019) for web platform features:
| browser | real values | true values |
null values |
|---|---|---|---|
| All 8 browsers | 58.82% | 25.08% | 16.10% |
| chrome | 68.05% | 25.27% | 6.68% |
| chrome android | 56.60% | 34.27% | 9.13% |
| edge | 59.18% | 19.39% | 21.44% |
| firefox | 75.34% | 14.86% | 9.80% |
| ie | 65.40% | 14.18% | 20.42% |
| safari | 51.11% | 24.83% | 24.07% |
| safari ios | 42.77% | 29.40% | 27.86% |
| webview android | 52.15% | 38.48% | 9.37% |