Test results are inaccessible to red-green color-blind, blind and CI

[JohnAlbin]

TLDR

I'm using AVA 4.0.0-rc.1 and have a few tests marked as ".failing" since they are known test failures. When looking at the test output, I expect that I can tell which tests are known failures, but known-failing tests look exactly like passing tests in some situations.

Details

AVA's default output doesn't clearly show which of the tests are known failures because it only distinguishes between test successes and test failures using color. When you only use color to distinguish between different things in an interface, you are making those things inaccessible to some people.

Here's some AVA test results on my terminal. You can see that both known failures and successes have the ✔ checkmark prepended and they only differ based on their color:

This is how those same AVA test results appear on GitHub's CI, which is similar to how red-green color-blind would see it. There's no way to tell which tests are known failures:

We need to re-think how these test results are displayed so they are accessible to everyone, including:

• developers who are red-green colorblind
• developers using screen readers
• developers who don't have a color-enabled terminal
• developers reading a plain text log file (like in a CI)

Possible solutions

As a reference, here's a screenshot of skipped, failing, passing and known failing tests:

Maybe known failures should be in "skipped" yellow with a ✖ cross instead of the checkmark. Known failures are more like skipped tests than anything else.

Also, right now failing tests only have a red ✖ with normal-colored descriptive text. Can we make the descriptive text of failing tests all in red? And maybe bold too? The bold isn't necessary since failures are printed again at the end of the test result list, but it helps the failures visually stick out from the rest of the list.

As for people who are using screen readers, I think we might need to add text to describe what -, ✖, ✔ mean. Especially if we use ✖ for known and unknown fails. Maybe this?:

  ✔ PASS: Example › should be an example test
  - SKIP: Example › should be an example test
  ✖ FAIL: Example › should be an example test
  ✖ KNOWN FAIL: Example › should be an example test (or maybe SKIP FAIL or SKIPPED FAIL)

Here's a mockup with color and bold added:

[novemberborn]

Thanks for the feedback @JohnAlbin!

Counter-intuitively known failures count as a success — it's only when they stop failing that there's a problem. It being hard to filter the output because the symbol is reused isn't necessarily a display issue.

I think we may be able to make some tweaks with the current implementation, though to really make strides I think we need to tackle https://github.com/avajs/ava/issues/2500.

[JohnAlbin]

Counter-intuitively known failures count as a success

Yep. It's one of the features I love about ava. Being able to document known bugs with tests makes it easer to fix them.

I actually noticed this issue because I have some known test failures in my project's integration tests with a dependency. I wanted to show the dependency project which tests were failing by showing them the GitHub Actions log results, BUT there's no color in the log files so it's impossible for anyone to read the the publicly-accessible log files and tell which tests are failing.

It being hard to filter the output because the symbol is reused isn't necessarily a display issue.

If we "add text to describe what -, ✖, ✔ mean", I'm fine with known failing tests still using ✔ icon because you can still tell that this red text is a known failure:

✔ KNOWN FAIL: Example › should be an example test (or maybe SKIP FAIL or SKIPPED FAIL)

Personally, I would still prefer the ✖, but I'm fine with the ✔. It's a passing test because it is failing and marked as .failing, so there's definitely ambiguity about what the "right" icon should be. :)