What to do with per-key Baseline statuses

[LeoMcA]

tl;dr I don't think per-key Baseline statuses are yet correct enough to use to power the Baseline banner at the top of MDN pages, and as a first step will merely use them as a replacement for the blocklist we currently manually maintain

Background

A short bit of background for those not familiar with the general issue (which can be skipped by those that are): Baseline statuses on MDN are currently calculated on a feature level (where something like "Async clipboard" is a feature), but this leads to some oddities where particular parts of a feature which are Baseline are displayed as not (or not widely available) because the feature as a whole isn't (despite it being entirely possible to e.g. use the writeText method of async clipboard without using the rest of the feature). So instead of displaying the overall status of the feature a BCD-key/page belongs to, we've received guidance from upstream to display per-BCD-key statuses, calculated with compute-baseline, at the top of pages.

The experiment

Before implementing this, I had some suspicions that there would still be some edge cases, so I wanted to investigate those. To do this, for every page on MDN which has a BCD key which is part of a web-features feature, I compared the compute-baseline BCD key status to the web-features feature status. Here's the results:

Pages with multiple BCD keys: 89 (I'll be auditing these soon and trying to map them directly to features, so let's ignore these for now)

Pages where the feature and key status are the same: 2307

This is good, and shows that for the vast majority of pages, whether we use the feature status or the per-BCD key status, we're ok.

Demotions

Pages where the feature is a higher status than the key (i.e. the page has been "demoted"): 32

• High to low: 8
• High to "not": 21
• Low to "not": 3

These demotions are a bit weird, I'm not sure why these keys are part of a feature when they don't contribute to the status. @ddbeck or @foolip perhaps you can shed some light? These 32 pages present a bit of a bug, as currently we show a Baseline banner on them when they're either only-recently usable, or not usable at all. So we need to do something here.

Promotions

Pages where the feature is a lower status than the key (i.e. the page has been "promoted"): 65

• Low to high: 42
• "Not" to high: 13
• "Not" to low: 10

Auditing these by page type, the vast majority seemed ok, and were "leaf" pages (e.g. interface properties or methods) which is exactly what we wanted to promote with per-key statuses (like the earlier example of writeText).

However, there were some page types which made me cautious, namely web-api-interface pages, and subpages of Navigator, since they tend to document the methods and properties on the interface as much as the interface itself - and those methods and properties may have a lower status. Let's look at specifically what these pages are:

• Interface pages
  • Low to high
    • /en-US/docs/Web/API/ByteLengthQueuingStrategy
      • seems fine, none of its methods/properties make the feature low
    • /en-US/docs/Web/API/Clipboard
      • not fine, some of its methods/properties are what make the feature low
    • /en-US/docs/Web/API/CountQueuingStrategy
      • seems fine, none of its methods/properties make the feature low
    • /en-US/docs/Web/API/InputEvent
      • maybe ok? one of its methods/properties make the feature low
    • /en-US/docs/Web/API/ReadableStream
      • maybe ok? two of its methods/properties make the feature low
  • Not to high
    • /en-US/docs/Web/API/HTMLDataListElement
      • not sure, is there value in knowing the js interface for an element works cross-browser if the element itself doesn't
  • Not to low
    • /en-US/docs/Web/API/RTCEncodedAudioFrame
      • seems fine, none of its methods/properties make the feature "not", though its constructor does
    • /en-US/docs/Web/API/RTCEncodedVideoFrame
      • seems fine, none of its methods/properties make the feature "not", though its constructor does
• Navigator pages
  • Low to high:
    • /en-US/docs/Web/API/Navigator/clipboard
      • Not ok, documents a method readText() which is low
  • Not to high:
    • /en-US/docs/Web/API/Navigator/mediaSession
      • Potentially ok

There were also some pages outside these pages types where the promotion seemed a little wrong, for instance the overflow-related pages. Here, obviously, overflow itself is Baseline high, but the clip functionality takes the feature into Baseline low, and it's difficult to decide what to display at the top of page: without otherwise warning that the clip functionality is Baseline low (with an inline status, for instance), I think it's misleading to overall call the page Baseline high.

The data

I've dug into the data above, but you might want to yourself. I present some that and some incredibly messy and inefficient code to regenerate the files if you so desire here: https://github.com/LeoMcA/yari/commit/74587d07f516153701952e6a0bcf42cc38e534f2

If you just want the data I used:

• https://github.com/LeoMcA/yari/blob/74587d07f516153701952e6a0bcf42cc38e534f2/baseline.count.json
• https://github.com/LeoMcA/yari/blob/74587d07f516153701952e6a0bcf42cc38e534f2/baseline.list.json

If you want to generate it yourself:

• Check out the branch
• yarn && yarn build --nohtml --locale en-us
• Wait about 5 minutes

To explain the files:

• Both baseline.count.json and baseline.list.json have roughly the same layout, until their leaves
• In terms of the keys:
  • multiple_keys refers to pages with multiple bcd keys, we don't do any further calculation on these, and simply break them down by page type
  • previously_filtered refers to pages in the current manually-maintained blocklist, we break these down by per-key status, and also include them again in the keys below
  • feature_{high|low|not} groups the pages by their web-features feature Baseline status, we break these down on their per-key status
  • {high|low|not} groups the pages by their compute-baseline per-key Baseline status, we break these down by page-type
  • all groups all the page types within a status, for easy counting
  • * (any other key) is the page type
• baseline.count.json then simply gives counts for the pages
• baseline.list.json lists our their urls

Conclusion

For now, I'm not comfortable simply showing the compute-baseline status for a key at the top of pages.

I want to experiment with traversing down the BCD tree for a key, and taking the lowest status for the result of compute-baseline for each of those - I reckon it may produce what we want.

But for now, given we have some clearly-wrong statuses (especially the "demotions" above), I'll implement conditional hiding of the banner if the web-feature status and compute-baseline status disagree, so we no longer need to manually curate the blocklist.

[ddbeck]

We had a conversation today about showing tentative banners. Here's a possible flowchart for deciding whether show/hide or caveat a banner:

Source in the D2 playground

[ddbeck]

An issue on web-features alerted me to one more asterisk-worthy case: when there's a split between desktop and mobile browsers of the same brand. This new flow chart handles that situation:

Source in the D2 playground

[ddbeck]

Here's a revised flowchart, handling the discouraged case (new question in orange)

Source in the D2 playground

[ddbeck]

One thing that this version of the chart doesn't cover is a compat key in multiple features, which is related to but not strictly dependent on the method used to select a banner.

If a compat key appears in more than one feature, the nodes "is the compat key in a web-features feature" and "collect table row statuses" are affected.

For "is the compat key in a web-features feature?" node, you'd have to search for the key across all features, finding all the headline statuses which apply to that key. If there was more than one, MDN would have to make an editorial decision:

1. Choose the least- or most-available headline status (i.e., report restrictively or generously), then continue with the flowchart.
2. Opt in to a specific feature for that page (e.g., set web-feature: some-id frontmatter for that page) and use its headline status explicitly, then continue with the flowchart.

(My inclination would be to do 2, but use the most-available headline status as a fallback if the frontmatter is unset.)

For "collect table row statuses" node, you'd have to search for the keys across all features, finding all the per-key statuses which apply to each key in the compat table. After that, you'd proceed as normal. Effectively, you'd pretend that each key with two or more statuses has extra (hidden) rows in the table.

[LeoMcA]

For "collect table row statuses" node, you'd have to search for the keys across all features, finding all the per-key statuses which apply to each key in the compat table. After that, you'd proceed as normal. Effectively, you'd pretend that each key with two or more statuses has extra (hidden) rows in the table.

Is the by_compat_key status influenced by the feature group status, or would it always be the same across features?

[ddbeck]

Is the by_compat_key status influenced by the feature group status, or would it always be the same across features?

Right now they are not influenced by the group status, but that's sort of an implementation detail. I wouldn't assume that to always be true.

In an ideal world, we'd report the each feature's minimally useful set of keys (i.e., the set of keys without which the feature is not functional) and set the status for those keys together. This would fix the problem of, for example, an interface being accidentally exposed early—it happens more often than you'd expect. 😬 But we haven't sorted out minor/subfeature addressing in web-features yet, which is a prerequisite.

[caugner]

@ddbeck Can you clarify what "headline status" refers to here:

Do any of the table row statuses differ with the headline status?

Is this the status of the first row in the table, or the overall feature status?

Context: On the ImageDecoder page, we show "Limited availability", but with a cross for "Firefox", although all the features in this BCD table are supported by Firefox (just not all the features that are part of the web feature group).

[ddbeck]

The headline status refers to the status of a feature in the web-features package (as opposed to the per-key statuses given by the extended JSON release artifact or a status computed per-key by compute-baseline).

[LeoMcA]

We've been showing per-key statuses, along with an asterisk, for a while now: there's still the occasional confusion, but broadly they seem to be understood, and our number of "the BCD table disagrees with the Baseline banner" issues has fallen dramatically.