w3c
Change unexpected behaviors to negative side effects and add a "is test blocked" question to test runner
This issue implements changes documented in w3c/aria-at#1043 and proposes a solution for w3c/aria-at#1216.
Change wording in reports, candidate review, and test runner results
for w3c/aria-at#1043, make the following changes to assertion wording for unexpected behaviors, now to be called, negative side effects.
| Assertion Form | Old Wording | New Wording |
|---|---|---|
| Assertion Statement | Other behaviors that create moderate negative impacts are not exhibited | Moderate negative side effects do not occur |
| Assertion Statement | Other behaviors that create severe negative impacts are not exhibited | Severe negative side effects do not occur |
| Assertion Phrase | Other behaviors that create moderate negative impacts are not exhibited | not cause moderate negative side effects |
| Assertion Phrase | Other behaviors that create severe negative impacts are not exhibited | not cause severe negative side effects |
Note: It is intentional that the assertion phrases start with lower case "not". This is part of the convention for the test format V2.
Change to test runner wording
for w3c/aria-at#1043, in the test runner, change wording as follows:
| Old Wording | New Wording |
|---|---|
| Were there additional undesirable behaviors? | Did negative side effects occur? |
| No, there were no additional undesirable behaviors. | No, negative side effects did not occur. |
| Yes, there were additional undesirable behaviors. | Yes, negative side effects occured. |
In the list of possible side effects, remove the words "behavior occurred" from each option. For example, change:
Reading cursor position changed in an unexpected manner behavior occurred
to:
Reading cursor position changed in an unexpected manner
In test runner, add question about ability to complete testing of the command
To resolve w3c/aria-at#1216, in the test runner, after the output and before the first assertion, add the following question. Default the answer to "No".
Did the response to COMMAND_NAME create conditions that invalidate assertions?
If the answer is yes:
- Disable assertion verditcs.
- Set the answer to negative side effects to "yes" and disable "No".
- On submit, check to make sure at least one side effect is set to "severe" for the command.
In reports, if the assertions were invalidated, instead of a verdict of "pass", "fail", "supported" or "unsupported", show a verdict of "Untestable". When this happens, the assertion for severe negative side effects should have a verdict of "failed". The assertion for moderate negative side effects could nbe "pass" or "fail". Note that it is possible that multiple side effects occurred and only one is severe.
Hi @mcking65 , this is marked as incomplete. Is the plan to discuss it during a CG meeting before moving forward with it ?
@ccanash
The description of this is now complete. Please expedite these changes. Without this feature, we cannot produce accurate reports for several test plans, including all the sliders. We will also update some VoiceOver reports, e.g., command button, once this feature is available.
The ARIA-AT Community Group just discussed App issue 1352 - Changes for negative side effects and untestable assertions.
The full IRC log of that discussion
<jugglinmike> Topic: App issue 1352 - Changes for negative side effects and untestable assertions<jugglinmike> github: https://github.com/w3c/aria-at-app/issues/1352
<jugglinmike> Matt_King: I put together a detailed plan that talks about the test runner and the reports
<jugglinmike> Matt_King: When you encounter a test like this (where it technically passed an assertion but for an inappropriate reason)
<jugglinmike> Matt_King: ...there would be a checkbox which indicates whether the assertions were testable
<jugglinmike> Joe_Humbert: So if it doesn't work as we think it should (it skips over or it says nothing at all, for example), there will be a test box for us to use which says, "we can't apply these assertions"
<jugglinmike> Matt_King: [summarizes the full UI and workflow as proposed in the issue]
<jugglinmike> Matt_King: I would like this to be expedited as quickly as possible so that we can get accurate reports on all of the sliders. I think we may even need to re-run a few VoiceOver tests because we encountered this problem and the way we reported them to Apple was confusing
<jugglinmike> Carmen: Sounds good. We have a planning meeting tomorrow, so we can prioritize this work accordingly
<jugglinmike> Hadi: How often is this condition?
<jugglinmike> Matt_King: It occurred on any test plan with Shift+J for one screen reader. We also just found it in a class of tests for NVDA
<jugglinmike> Matt_King: So far, it's happened in probably seven or eight test plans and with two screen readers
<jugglinmike> Matt_King: We discussed about what to do without this new feature. We could just mark assertions as failing, but that gives a misleading picture of what is wrong. It produces confusing data, and I don't think we want that
Hi @mcking65! My work implementing this feature has uncovered a few edge-cases that could use your input.
In reports for commands with unexpected behaviors, change the phrase that appears above the table of unexpected behaviors from:
Other behaviors that create negative impact:
to:
Negative side effects of COMMAND_NAME
and mark it up as a level 4 heading (same level as the command results heading)
Two questions about this:
- Alternate versions of this component appear on the Candidate Test Plan Review page and also on the Test Run page (when the test run has been completed). Would you to insert a new heading element in those other two contexts? If so, can you give guidance on the desired heading levels for those cases (as they both omit the "command results" heading)?
- When there is no table, the application currently displays the text "none" here. Would you like that word (and the colon which precedes it) to be part of this new level 4 heading? Or would you like the word "None" to appear as a separate text node immediately following the new level 4 heading?
In the list of possible side effects, remove the words "behavior occurred" from each option. For example, change:
Reading cursor position changed in an unexpected manner behavior occurred
to:
Reading cursor position changed in an unexpected manner
Does this also go for the final option, "Other behavior occurred"? That is to say: would you like the final option to simply read, "Other" instead?
If the checkbox is checked:
- Disable assertion verdict radio buttons.
- In the back-end, record verdicts as "Untestable"
- Set the answer to the negative side effects question to "yes" and disable "No".
- On submit, check to make sure at least one side effect is set to "severe" for the command.
Manually selecting "yes" causes the fieldset titled "Undesirable behaviors" to appear. Should that fieldset also become visible when the proposed "untestable" checkbox is checked? If so, should the nested checkboxes be left unchecked (as per the current behavior) or automatically manipulated in some way (perhaps checking "Other")?
In reports for commands with unexpected behaviors, change the phrase that appears above the table of unexpected behaviors from:
Other behaviors that create negative impact:
to:
Negative side effects of COMMAND_NAME
and mark it up as a level 4 heading (same level as the command results heading)
Two questions about this:
- Alternate versions of this component appear on the Candidate Test Plan Review page and also on the Test Run page (when the test run has been completed). Would you to insert a new heading element in those other two contexts? If so, can you give guidance on the desired heading levels for those cases (as they both omit the "command results" heading)?
First, I think there may be a point of confusion. When I said "command results heading", I was not referring to the level 3 heading with text "Results for each command". I was referring to the level 4 headings with form "f (virtual cursor active) Results: 5 passed, 0 failed, 0 unsupported". This heading is available in all contexts. However, in the test runner and candidate review, it is a level 3, not a level 4.
When looking at the JAWS candidate review experience for test 1 in the Action Menu Button Example test plan, in the "Test Results for Chrome" disclosure, there is a level 3 heading with content "b (virtual cursor active) Results: 5 passed, 0 failed, 0 unsupported". This is the command results heading I referenced in the spec.
On the candidate review page, If this command caused negative side effects, then add a level 3 heading after the command results table with "Negative side effects of b (virtual cursor active)". Note that this would be a level 4 if it were in the context of a report page.
- When there is no table, the application currently displays the text "none" here. Would you like that word (and the colon which precedes it) to be part of this new level 4 heading? Or would you like the word "None" to appear as a separate text node immediately following the new level 4 heading?
If there were no negative side effects, just show the text, no heading, e.g., "Negative side effects of b (virtual cursor active): None"
Even better would be a clear statement: "The command 'b (virtual cursor active)' did not cause any negative side effects."
In the list of possible side effects, remove the words "behavior occurred" from each option. For example, change:
Reading cursor position changed in an unexpected manner behavior occurred
to:
Reading cursor position changed in an unexpected manner
Does this also go for the final option, "Other behavior occurred"? That is to say: would you like the final option to simply read, "Other" instead?
Yes.
If the checkbox is checked:
- Disable assertion verdict radio buttons.
- In the back-end, record verdicts as "Untestable"
- Set the answer to the negative side effects question to "yes" and disable "No".
- On submit, check to make sure at least one side effect is set to "severe" for the command.
Manually selecting "yes" causes the fieldset titled "Undesirable behaviors" to appear. Should that fieldset also become visible when the proposed "untestable" checkbox is checked?
Yes.
If so, should the nested checkboxes be left unchecked (as per the current behavior) or automatically manipulated in some way (perhaps checking "Other")?
Leave them unchecked; he user must choose.
@ccanash @howard-e
After generating a report and assessing understandability, I think we need to change two of the reporting requirements above. I don't think these changes need to block deployment, but I think we need to make these changes within days of deployment if possible.
I raised issue #1429 to describe changes to the following two requirements:
Update reporting code to support untestable assertions
When assertions have verdicts of "Untestable", change the reports to: 4. When calculating the percentage of assertions that pass, leave untestable assertions out of both numerator and denominator. 7. Even though untestable assertions are not technically "filures", add rows for them to the summary of failures.
This was released to production. Improvements to this will be tracked in #1429 and #1427