Guidelines for associating tests with features

[foolip]

trafficstars

In https://github.com/web-platform-tests/rfcs/blob/master/rfcs/web_features.md we introduced the ability to associate tests with features using WEB_FEATURES.yml files, and showing them on wpt.fyi like this:

• https://wpt.fyi/results/?label=master&label=experimental&aligned&q=feature%3Acanvas-text-baselines
• https://wpt.fyi/results/?label=master&label=experimental&aligned&q=feature%3Afont-palette
• https://wpt.fyi/results/?label=master&label=experimental&aligned&q=feature%3Anesting

There are now over 150 features mapped using WEB_FEATURES.yml files. To allow more people to contribute to this, it's clear that some guidelines will be needed.

Questions and tentative answers:

• When should a test be linked to a feature?
  • When it's in the scope of the feature in web-features judging by its description and compat_features list.
  • Tests should match the spec, but contributors aren't expected to verify this, assume the initial review covered this. If it becomes known that a test doesn't match the spec, the best course of action is probably to make it tentative.
• Can a test be linked to multiple features?
  • Yes! But this is only useful when two features ship in different orders in different browsers, to allow filtering out tests that cover a yet-to-be-implemented feature.
• How can a specific test be excluded?
  • There's no syntax for this, but the need is obvious. Right now the only way is to list all tests and remove some.
• What about tentative tests?
  • It's fine if they're captured by the patterns, they can be excluded using !is:tentative on wpt.fyi if desired.
• What about subtests?
  • There's no way to associate only a few subtests with a feature. This hasn't been a big problem in practice, but in a few cases this makes it impossible to capture all the tests.

Beyond this, some form of test suite review by feature experts would be useful, but I'll file a separate issue about that when I've thought through which repo it belongs in.

cc @jcscottiii

[foolip]

Thinking about where documentation for this would be discoverable, at minimum it should show up in https://web-platform-tests.org/search.html?q=web_features. It doesn't fit cleanly in any existing section in https://web-platform-tests.org/#table-of-contents, but under "Writing Tests" probably makes the most sense.