project icon indicating copy to clipboard operation
project copied to clipboard
verified publisher icon openwebdocs

OWD project: Fix errors in BCD for API, CSS and JS going back to at least 2020

Open foolip opened this issue 3 years ago • 10 comments

https://github.com/openwebdocs/project/issues/66 and similar efforts in the past have eliminated all true and null values in BCD for API, CSS and JS data. @queengooborg and I used https://mdn-bcd-collector.appspot.com/ to finish the API data over the course of 2021, and the collector also covers a lot of CSS and JS.

We have found that comparing BCD to feature detection results invariably finds a lot of errors in BCD. By comparing BCD with the collector results, we can find many false positives and false negatives. This will likely lead to many features looking "better" (more green) as support has shipped in the past 2 years but BCD was never updated.

Such cross-checking can in principle go back over 10 years, but I suggest limiting it to browser releases after 2020-01-01, and updating earlier data only when it's really easy and does not require much digging to verify.

foolip avatar Feb 14 '22 08:02 foolip

@queengooborg @robnyman do you have permissions to label this as a proposal for Q3 planning?

foolip avatar Jun 22 '22 13:06 foolip

I have a feeling that this will span across both Q3 and Q4 for how big the project is. Maybe we should separate this into three projects, one for each category?

queengooborg avatar Jul 30 '22 20:07 queengooborg

I have a feeling that this will span across both Q3 and Q4 for how big the project is. Maybe we should separate this into three projects, one for each category?

Yes, I think having smaller projects is a good thing here. This helps get focused and claiming victories is always a good feeling and a good way to communicate progress.

teoli2003 avatar Jul 31 '22 06:07 teoli2003

I think an important first step here is to list all of the contradictions between BCD and collector data for the past two years' releases. There is no script for this currently.

foolip avatar Jul 31 '22 11:07 foolip

Thanks for filing this OWD project proposal! I think this proposal is specific and actionable. We need to determine the priority relative to all other projects, though.

The OWD prioritization criteria guides us to make decisions. Please complete and/or correct the following, so we can judge the priority correctly here.

Effort

It seem like there is some scripting needed to determine the contradictions between BCD and the collector. This will likely produce a list of some easy cases and cases where it is unclear what is correct. It probably requires expert knowledge to make the correct calls. The above comments suggest that this project could be split into parts to lower that workload.

Dependencies

  • Knowledge about BCD and the collector.
  • A script to list contradictions
  • Knowledge how to decide what is correct.

Community enablement

Correct me if I'm wrong but dealing with the contradictions is hard. Not sure how much we will be able to involve the community with this project.

Momentum

Seems like correct compat data is of interest for current initiatives like Interop?

Enabling learners

n/a Not part of any curriculum or learning pathway.

Enabling professionals

n/a

Underrepresented topics / ethical web

n/a

Operational necessities

n/a

Addressing needs of the Web industry

Interop?

Elchi3 avatar Sep 05 '22 14:09 Elchi3

In yesterday's planning call, @queengooborg said:

  • It probably makes sense to have different tasks for different content areas here (CSS, JS, APIs)
  • Vinyl reckons this is mostly done for Web APIs already.

Assigning this to @queengooborg to post an initial assessment of what work is required here and whether we want this project to be split up which would then allow us to prioritize say CSS data over other data.

Elchi3 avatar Sep 15 '22 13:09 Elchi3

To quote myself, I think we need a new approach to judge if this is done, and how much remains:

I think an important first step here is to list all of the contradictions between BCD and collector data for the past two years' releases. There is no script for this currently.

Just running the update-bcd script in the collector using data going back to 2020 and burning down all the changes can still leave many errors in the data. That's because of a number of TODOs in https://github.com/foolip/mdn-bcd-collector/blob/main/update-bcd.ts in non-trivial cases.

We can detect contradictions without being able to fix them automatically.

foolip avatar Sep 15 '22 14:09 foolip

https://github.com/foolip/mdn-bcd-collector/pull/2317 demonstrates many problems in update-bcd.

foolip avatar Sep 16 '22 07:09 foolip

In today's BCD call I proposed to a have "preparation" project for this issue.

I'd like to propose to do a project that creates a continuously updated BCD issue very similar to https://github.com/mdn/browser-compat-data/issues/3555.

In that issue, the collector reports something like this after every BCD release (percentages are fiction):

browser collector verified data collector reports wrong data collector is broken
total 50% 25% 25%
chrome 50% 25% 25%
firefox 50% 25% 25%
safari 50% 25% 25%

It only takes into account data for browser releases after 2020-01-01. We can see for which folders it should run. (api, js, css, mathml, svg)

@queengooborg what do you think about doing that in q4? It would involve work on the collector to spit out such metrics.

Once we have this, we follow up with projects to decrease the number of cases where the collector is broken and where the collector reports wrong data that probably needs fixing in BCD.

@foolip I hope this makes sense. Let me know if you have anything to add to this idea.

Elchi3 avatar Sep 21 '22 17:09 Elchi3

@Elchi3 that approach to burning down errors makes sense to me, it was very motivating in past efforts to see the numbers slowly approach 100%.

I think there are two numbers we can track:

  • Coverage of the collector tests of BCD. This is the inverse of what npm run find-missing-features reports, which currently reports 14% of features are missing. We might want to distinguish behavior features (with underscores) from the rest here. We should have 100% of non-behavioral features in APIs, CSS and JS covered.
  • Agreement between collector results and BCD. To avoid complexity/error in the computation itself, this should just compare the results for a single browser release to what BCD says about that release. So the maximum score is the number of browser releases since 2020 multiplied by the number of features in BCD the collector covers.

I don't think we can track where the collector is wrong in any meaningful way, we update custom tests continuously when we discover this.

foolip avatar Sep 22 '22 15:09 foolip

In our planning call yesterday, @queengooborg said that this is done. The statistics part / burn down issue that Philip and I talked about in here last hasn't implemented, though. Should we split that into a separate issue?

Elchi3 avatar Dec 15 '22 16:12 Elchi3

I think we should move it to a separate issue, yes -- perhaps open one up in the collector repository?

queengooborg avatar Dec 15 '22 18:12 queengooborg

It looks like there's quite a lot of differences between BCD and the collector remaining for post-2020 data. Here's what I did. First, list the browser releases that happened in 2020-2022 (ignoring 2023 to not make this a moving target):

  • Chrome 80-108
  • Firefox 72-108
  • Safari 13.1-16.1

Then, get the results for those from https://github.com/GooborgStudios/mdn-bcd-results/tree/01f943d878fc1d51364a2791906aefb69ade3692:

140 JSON files

7.1.2-chrome-100.0.4896.127-mac-os-10.15.7-ffe2e9031b.json 7.1.2-chrome-100.0.4896.127-windows-10-8998eec00e.json 7.1.2-chrome-101.0.4951.41-mac-os-10.15.7-8a1cfe58fd.json 7.1.2-chrome-101.0.4951.41-windows-10-d2716fd74c.json 7.1.2-chrome-102.0.5005.61-mac-os-10.15.7-d9e8fb093a.json 7.1.2-chrome-102.0.5005.63-windows-10-f7592ea982.json 7.1.2-chrome-103.0.5060.114-mac-os-10.15.7-85c9391558.json 7.1.2-chrome-103.0.5060.114-windows-10-12a6687f7d.json 7.1.2-chrome-104.0.5112.79-mac-os-10.15.7-299c5b8ca6.json 7.1.2-chrome-104.0.5112.81-windows-10-a8c7830090.json 7.1.2-chrome-105.0.0.0-mac-os-10.15.7-34efe1dbad.json 7.1.2-chrome-105.0.0.0-windows-10-dcf1d4f404.json 7.1.2-chrome-106.0.0.0-mac-os-10.15.7-45aec0e87b.json 7.1.2-chrome-106.0.0.0-windows-10-f619773e3f.json 7.1.2-chrome-107.0.0.0-mac-os-10.15.7-e1a00b13b0.json 7.1.2-chrome-107.0.0.0-windows-10-3b59b002b2.json 7.1.2-chrome-108.0.0.0-mac-os-10.15.7-9b04febc50.json 7.1.2-chrome-108.0.0.0-windows-10-911a38603a.json 7.1.2-chrome-80.0.3987.132-mac-os-10.16.0-420cbde938.json 7.1.2-chrome-80.0.3987.132-windows-10-9f300787d0.json 7.1.2-chrome-81.0.4044.129-mac-os-10.16.0-89dd32c263.json 7.1.2-chrome-81.0.4044.129-windows-10-de8828617c.json 7.1.2-chrome-83.0.4103.61-mac-os-10.16.0-2b2546d89a.json 7.1.2-chrome-83.0.4103.61-windows-10-9969adbbc5.json 7.1.2-chrome-84.0.4147.89-mac-os-10.16.0-4be183762c.json 7.1.2-chrome-84.0.4147.89-windows-10-9327d18402.json 7.1.2-chrome-85.0.4183.83-mac-os-10.16.0-64b24f8450.json 7.1.2-chrome-85.0.4183.83-windows-10-dd7f8b10eb.json 7.1.2-chrome-86.0.4240.183-mac-os-11.0.0-7b94a2415f.json 7.1.2-chrome-86.0.4240.183-windows-10-ea3e07ff5b.json 7.1.2-chrome-87.0.4280.66-mac-os-11.0.0-f286d06aef.json 7.1.2-chrome-87.0.4280.66-windows-10-def95c3fcb.json 7.1.2-chrome-88.0.4324.150-mac-os-12.6.1-f00f9a880d.json 7.1.2-chrome-88.0.4324.150-windows-10-ab92cc7b5c.json 7.1.2-chrome-89.0.4389.90-mac-os-12.5.1-7e5f79dc6e.json 7.1.2-chrome-89.0.4389.90-windows-10-e3b098eb52.json 7.1.2-chrome-90.0.4430.72-mac-os-10.15.7-ef6e1ffa88.json 7.1.2-chrome-90.0.4430.72-windows-10-16e9533da4.json 7.1.2-chrome-91.0.4472.106-mac-os-10.15.7-b081eb6770.json 7.1.2-chrome-91.0.4472.106-windows-10-0e27c41a16.json 7.1.2-chrome-92.0.4515.107-mac-os-10.15.7-5ab49ff904.json 7.1.2-chrome-92.0.4515.107-windows-10-beca2cb6a6.json 7.1.2-chrome-93.0.4577.63-mac-os-10.15.7-7ccbcfd007.json 7.1.2-chrome-93.0.4577.63-windows-10-a11ce93763.json 7.1.2-chrome-94.0.4606.71-mac-os-10.15.7-87d01e2c02.json 7.1.2-chrome-94.0.4606.71-windows-10-030848a705.json 7.1.2-chrome-95.0.4638.54-mac-os-10.15.7-d725413668.json 7.1.2-chrome-95.0.4638.54-windows-10-fd1c557e84.json 7.1.2-chrome-96.0.4664.110-mac-os-10.15.7-c3b8b140c0.json 7.1.2-chrome-96.0.4664.110-windows-10-11a22fbe1a.json 7.1.2-chrome-97.0.4692.71-mac-os-10.15.7-a9fa5d0ffb.json 7.1.2-chrome-97.0.4692.71-windows-10-f0bc1aa4a2.json 7.1.2-chrome-98.0.4758.80-mac-os-10.15.7-1745d784a3.json 7.1.2-chrome-98.0.4758.80-windows-10-a3cbe4a57a.json 7.1.2-chrome-99.0.4844.51-mac-os-10.15.7-f470137426.json 7.1.2-chrome-99.0.4844.51-windows-10-7d9f47b17b.json 7.1.2-firefox-100.0-mac-os-10.15-ec6711939d.json 7.1.2-firefox-100.0-windows-10-c72941eb1d.json 7.1.2-firefox-101.0-mac-os-10.15-8975a93263.json 7.1.2-firefox-101.0-windows-10-6d3bb1a67f.json 7.1.2-firefox-102.0-mac-os-10.15-e5c418ab14.json 7.1.2-firefox-102.0-windows-10-f6a0f0e410.json 7.1.2-firefox-103.0-mac-os-10.15-3d1c213f7b.json 7.1.2-firefox-103.0-windows-10-5ea1a7bea4.json 7.1.2-firefox-104.0-mac-os-10.15-fc437aaedb.json 7.1.2-firefox-104.0-windows-10-b5736c192f.json 7.1.2-firefox-105.0-mac-os-10.15-c656d893d8.json 7.1.2-firefox-105.0-windows-10-232eca12df.json 7.1.2-firefox-106.0-mac-os-10.15-02f2647e38.json 7.1.2-firefox-106.0-windows-10-685eea7f2b.json 7.1.2-firefox-107.0-mac-os-10.15-9936c4b4eb.json 7.1.2-firefox-107.0-windows-10-9e50eacb45.json 7.1.2-firefox-108.0-mac-os-10.15-078e58850b.json 7.1.2-firefox-108.0-windows-10-1b541f6e19.json 7.1.2-firefox-72.0-mac-os-10.0-d952a56c8f.json 7.1.2-firefox-72.0-windows-10-73bfa9f057.json 7.1.2-firefox-73.0-mac-os-10.0-3f6ed30904.json 7.1.2-firefox-73.0-windows-10-f517ff8f88.json 7.1.2-firefox-74.0-mac-os-10.0-371de8d7bb.json 7.1.2-firefox-74.0-windows-10-7aa6da912e.json 7.1.2-firefox-75.0-mac-os-10.0-2b1ea8024c.json 7.1.2-firefox-75.0-windows-10-e32f98d953.json 7.1.2-firefox-76.0-mac-os-10.0-0455ff975f.json 7.1.2-firefox-76.0-windows-10-9abac7c8d5.json 7.1.2-firefox-77.0-mac-os-10.0-20f1e5e76f.json 7.1.2-firefox-77.0-windows-10-7715e46944.json 7.1.2-firefox-78.0-mac-os-10.16-8bdb25e390.json 7.1.2-firefox-78.0-windows-10-b70f54b86d.json 7.1.2-firefox-79.0-mac-os-10.16-5b17c78136.json 7.1.2-firefox-79.0-windows-10-3dcd26cada.json 7.1.2-firefox-80.0-mac-os-10.16-04ad75c4f9.json 7.1.2-firefox-80.0-windows-10-58a961f38d.json 7.1.2-firefox-81.0-mac-os-10.16-5797eb08ed.json 7.1.2-firefox-81.0-windows-10-aa52916bf6.json 7.1.2-firefox-82.0-mac-os-10.16-c1951442cf.json 7.1.2-firefox-82.0-windows-10-3ba1bf157f.json 7.1.2-firefox-83.0-mac-os-10.16-350251a303.json 7.1.2-firefox-83.0-windows-10-de7147f017.json 7.1.2-firefox-84.0-mac-os-10.16-3d690475cf.json 7.1.2-firefox-84.0-windows-10-c6c155ddc2.json 7.1.2-firefox-85.0-mac-os-10.16-34aa5d36a4.json 7.1.2-firefox-85.0-windows-10-69c8eade53.json 7.1.2-firefox-86.0-mac-os-10.16-41d4dad35a.json 7.1.2-firefox-86.0-windows-10-ccb799ab46.json 7.1.2-firefox-87.0-mac-os-10.15-77861c77e9.json 7.1.2-firefox-87.0-windows-10-48e1fa95e5.json 7.1.2-firefox-88.0-mac-os-10.15-e7f0aadd30.json 7.1.2-firefox-88.0-windows-10-e5b7f549b1.json 7.1.2-firefox-89.0-mac-os-10.15-b447840869.json 7.1.2-firefox-89.0-windows-10-634f4f93a7.json 7.1.2-firefox-90.0-mac-os-10.15-e242b8521e.json 7.1.2-firefox-90.0-windows-10-9d7231d4a4.json 7.1.2-firefox-92.0-mac-os-10.15-ec9fcd0d77.json 7.1.2-firefox-92.0-windows-10-32f2e0b167.json 7.1.2-firefox-93.0-mac-os-10.15-993ec20b5d.json 7.1.2-firefox-93.0-windows-10-2dba5d6ba1.json 7.1.2-firefox-94.0-mac-os-10.15-134927efa8.json 7.1.2-firefox-94.0-windows-10-7e516e9a09.json 7.1.2-firefox-95.0-mac-os-10.15-23e3352e17.json 7.1.2-firefox-95.0-windows-10-3ad897ac33.json 7.1.2-firefox-96.0-mac-os-10.15-f79f76076a.json 7.1.2-firefox-96.0-windows-10-6881bc118d.json 7.1.2-firefox-97.0-mac-os-10.15-c95bde2776.json 7.1.2-firefox-97.0-windows-10-2ea6192280.json 7.1.2-firefox-98.0-mac-os-10.15-9b1549d8f9.json 7.1.2-firefox-98.0-windows-10-9519c8fa95.json 7.1.2-firefox-99.0-mac-os-10.15-b33386cca3.json 7.1.2-firefox-99.0-windows-10-8b83dccdc0.json 7.1.2-safari-13.1.2-mac-os-10.15.6-c6025b2d51.json 7.1.2-safari-14.1-mac-os-10.15.7-f58cb3d759.json 7.1.2-safari-14.1.2-mac-os-10.15.7-23b8baed87.json 7.1.2-safari-15.1-mac-os-10.15.7-0404d9f8ea.json 7.1.2-safari-15.2-mac-os-10.15.7-57519637a3.json 7.1.2-safari-15.3-mac-os-10.15.7-dcb079f647.json 7.1.2-safari-15.4-mac-os-10.15.7-ed8181bc2c.json 7.1.2-safari-15.5-mac-os-10.15.7-c73e787a42.json 7.1.2-safari-15.6.1-mac-os-10.15.7-7e1e00c5a8.json 7.1.2-safari-15.6.1-mac-os-10.15.7-83d41e8e05.json 7.1.2-safari-16.0-mac-os-10.15.7-f0ead7402e.json 7.1.2-safari-16.1-mac-os-10.15.7-1a469448f4.json

Then, running npm run update-bcd from https://github.com/GooborgStudios/mdn-bcd-collector (with no arguments other than the reports) to see what changes it makes.

https://github.com/mdn/browser-compat-data/pull/18539 shows changes to 220 files.

After differences for Chrome, Firefox and Safari desktop are sorted out, the next step would be to see what additional errors exist for Edge (likely many) and mobile browsers.

Is this project continuing in Q1? My definition of done would be that npm run update-bcd doesn't produce any changes when given 2020-2022 releases from all browsers in BCD.

foolip avatar Jan 05 '23 04:01 foolip

In our planning call yesterday, @queengooborg said that this is done.

After fixing a bug in the update-bcd script regarding truthy->false changes, and after fixing a number of tests, I rescind my statement that this project is done. There's actually a significant amount of changes that still need to be made to synchronize the collector's results and BCD. Running the following commands produces the following number of touched files:

  • npm run update-bcd -- -b chrome -b chrome_android -b edge -b webview_android -r 80-108 (Chromium): 88 changed files
  • npm run update-bcd -- -b firefox -b firefox_android -r 72-108 (Firefox): 52 changed files
  • npm run update-bcd -- -b safari -b safari_ios -r 13.1-16.2 (Safari): 62 changed files

(Note: the -r/--release argument filters for changes specific to that version of the browser.)

These numbers don't include changes that set the browser to false, either. There are a significant number of those to check through as well. Although a number may simply be test issues, known collector limitations or other small issues, they will still need to be carefully checked.

By the way, may we break this project up into smaller chunks? Perhaps, divided into each category and each year (ex. API 2020, JS 2022, etc.)?

queengooborg avatar Jan 05 '23 06:01 queengooborg

Breaking this into smaller chunks sounds great, it's a lot of work, and for reviewers it's probably easier to review if there's a theme of the day, rather than simply going A-Z.

I think the highest priority should probably be where a truthy value changes to false or vice versa, because that means BCD (or the collector) is wrong about current support. If update-bcd works like I think it should, that can probably be done by running the collector with just the latest release until it produces no changes.

No doubt there will be a lot of bugs to fix in the tests and some in update-bcd as well, so it'll be important that there's someone who can review such changes. Reviewer bandwidth in BCD is also important, in my experience these reviews are far from trivial, it's only trivial if you assume the collector is correct which is not a safe assumption.

@queengooborg do you think "changed files" is an OK way to track this, or how would you ideally want to define the burndown list and split into manageable chunks?

foolip avatar Jan 05 '23 08:01 foolip

I pushed a commit that adds the ability to specify false as a value (-r false), which will filter to only truthy->false changes. I agree this would be helpful to run through first, and then run through smaller ranges of changes per browser until all changes have been verified as accurate or fixed/overridden in the collector.

I opted to specify "changed files" as the metric in my report above as it was the easiest for me to quickly pull numbers for, but the best metric would probably be the diff count, since some files may have many more changes than others (especially bigger interfaces like Document and Element). I'm also thinking that the best way to split this would be per category and browser (ex. API: Chrome), with project tasks for each grouping of releases (ex. 2022 releases, 2021, falsy...). It would look something like this:

  • API
    • Chrome [Project]
      • Falsy [Project Task]
      • 2022 Releases [Project Task]
      • 2021 Releases [Project Task]
      • ...
    • Firefox [Project]
      • ...
    • Safari [Project]
      • ...
  • CSS
    • ...
  • JS
    • ...

queengooborg avatar Jan 05 '23 08:01 queengooborg

That breakdown looks pretty good to me!

My assumption is that after Falsy + 2022 Releases + 2021 Releases, the kind of exercise I did in https://github.com/openwebdocs/project/issues/85#issuecomment-1371759747 would produce no changes at all given the post-2020 results. In other words, the only remaining changes that the collector could make should be ones about pre-2020 differences, that it would need pre-2020 data to know about. I point this out because there's a difference between "update BCD to reflect all additions/removal in release X" and "update BCD to be consistent with release X", and it's the latter I have in mind for this project.

foolip avatar Jan 05 '23 12:01 foolip

fwiw https://github.com/mdn/browser-compat-data/issues/19188 is the specific request to generate statistics for this goal.

Elchi3 avatar Apr 05 '23 09:04 Elchi3

We have been using https://github.com/mdn/browser-compat-data/pull/19585 as a tracker and now reached 100 files of discrepancies. Working on getting these resolved, too.

Elchi3 avatar Jun 05 '23 13:06 Elchi3

At 60 files now. Aiming to finish this by the end of the quarter.

Elchi3 avatar Jun 13 '23 09:06 Elchi3

We have finished this goal as of today! Whoo!

There are a few interfaces that were skipped: PushManager and PushSubscription (these two require an overhaul of the collector's code, which isn't worth it at this stage), as well as RTCStatsReport (which requires a WebRTC expert to verify our tests).

queengooborg avatar Jun 27 '23 09:06 queengooborg

This is fantastic news, thank you @queengooborg for all your hard work on this!

foolip avatar Jun 28 '23 04:06 foolip

See https://github.com/openwebdocs/project/issues/167#issuecomment-1610194655 for how large the efforts would be if we would go further into the past.

Elchi3 avatar Jun 28 '23 09:06 Elchi3