Summary of failing assertions miscalculates number of commands

[mcking65]

In the section of reports that summarizes failing assertions, the number of commands is incorrect. For example, in VoiceOver for macOS and Safari for Command Button Example | ARIA-AT Reports, the sentence that summarizes the failures after the heading is:

3 assertions failed across 10 commands in 1 tests.

There is only one command, not ten commands, that failed, "shift+j".

[howard-e]

Thanks @mcking65. This issue is currently being addressed in #1358. A discussion point came up around what is considered a unique command when used across multiples tests, so we wanted to remove any further ambiguity.

Consider the following summary of failing assertions table where 2 different tests, with 3 different assertions are all using the same command:

Test Name	Command	Assertion Priority	Assertion	JAWS Response
Navigate backwards to a button	Shift+j (virtual cursor active)	MUST	assertionStatement1	response1
Navigate backwards to a button	Shift+j (virtual cursor active)	SHOULD	assertionStatement2	response2
Navigate forwards to a button	Shift+j (virtual cursor active)	MUST	assertionStatement3	response3

So, should the table state: a) "3 assertions failed across 2 commands in 2 tests"? OR b) "3 assertions failed for 1 command in 2 tests"?

a) is what happens the patch is doing now.

cc @stalgiag

[howard-e]

@stalgiag had a sync discussion with @mcking65 where we went over my above comment. Found that several variations of reworking that text may potentially be confusing to viewers so it made sense to isolate the metrics (and revisit if any feedback arises from that). So the new format would be:

x of y total assertions fail across:
* z1 tests
* z2 unique assertion statements
* z3 unique commands

So for the linked report in the top comment (Command Button Example V24.03.12 with VoiceOver for macOS and Safari), this would read:

3 of 40 total assertions fail across:
* 1 test
* 3 unique assertion statements
* 1 unique command

[stalgiag]

@stalgiag had a sync discussion with @mcking65 where we went over my above comment. Found that several variations of reworking that text may potentially be confusing to viewers so it made sense to isolate the metrics (and revisit if any feedback arises from that). So the new format would be:

x of y total assertions fail across:
* z1 tests
* z2 unique assertion statements
* z3 unique commands

So for the linked report in the top comment (Command Button Example V24.03.12 with VoiceOver for macOS and Safari), this would read:

3 of 40 total assertions fail across:
* 1 test
* 3 unique assertion statements
* 1 unique command

Is your understanding that this subheading should now be a subheading followed by an unordered list? This could have broader UI consequences.

[mcking65]

@stalgiag

This part of the text:

x of y total assertions fail across:

would not be a heading, just a paragraph element followed by a list of 3 items.