ADO extension: baselining might be too sensitive to unrelated changes

[dbjorge]

The baseline support added to the ADO extension v1.1.0 is sometimes overly sensitive to non-accessibility-related changes in the page. In particular, if a baselined failure's axe-core-suggested HTML snippet ends up changing due to a PR in a way that doesn't impact whether it fails an accessibility rule or not, the baselining logic will currently see that as "one failure fixed, one new failure found", since it will consider the versions of the failure with different snippets as different.

This tradeoff was made intentionally in the interest of having baselining's idea of failure grouping be consistent with how failure grouping works already in accessibility-insights-service, but it's something that we predict might end up being unexpected for users. We don't have good data on exactly how often this is going to be hit in practice, so we'll use this tracking issue as a place to track potential user feedback on this topic.

If you're a user reading this that is frustrated by the behavior, we'd love if you could upvote this issue (with a thumbs-up reaction) and leave a comment letting us know the specific example that caused you trouble! (ideally, including a diff of the relevant section of your baseline)

[msftbot[bot]]

This issue has been marked as ready for team triage; we will triage it in our weekly review and update the issue. Thank you for contributing to Accessibility Insights!

[asksep]

The Accessibility Insights team has received several upvotes for this issue over the course of several months, and will be investigating a solution.

[brocktaylor7]

Problematic Scenario

The most common scenario that we have come across that causes this issue is CSS selectors being auto-generated in a non-deterministic manner by webpack or other build processes.

For example, an element and CSS class may look like this in the source code:

<p class="test-class">Hello World</p>

And once built, the build process will append a random string to the end of the class (-Gz1t in this case)

<p class="test-class-Gz1t">Hello World</p>

This randomly generated string changes on every build, causing mismatches in the baseline file each time.

---

Possible Solutions/Workarounds

Adding an id/class/attribute to uniquely identify problematic element

Axe, the accessibility scanning engine used by Accessibility Insights, uses a hierarchy of possible selectors to uniquely represent an element. If a unique selector is found on any step of the hierarchy it will use that selector and skip the rest of the hierarchy.

The hierarchy used is as follows:

1. If an element has an id which is unique with the page
2. Class names which are unique within the page
3. Custom attributes which are unique with in the page
4. Custom complex selector using classes and attributes of the element and it's ancestors

If you're able to give the problematic element(s) a unique id or css class name which is not modified on build, Axe can use that selector instead of the auto-generated one.

Modifying build strategy

Depending on your project and build setup, you may be able to use a different strategy for how the build process generates selector names. For example, our own webpack settings for our CSS modules uses something like this:

{
  loader: 'css-loader',
  options: {
    esModule: true,
    modules: {
      localIdentName: '[local]--[hash:base64:5]'),
    }
  }
}

This produces deterministic selector names as long as the contents of the individual CSS module in question are unchanged. This would still cause the baseline to break if the contents of the module are changed, but it would still be a drastic improvement over breaking every build even when no source changes occur.

[DaveTryon]

Unfortunately, this is out of scope given our current priorities.

Thanks for using Accessibility Insights!