w3c
Clarification Needed: Process for Handling VoiceOver/NVDA Bot Speech Output Feedback
A question has come up regarding how to handle feedback specifically concerning the speech output of the VoiceOver and NVDA automation bots used during testing.
Currently, observations about inconsistencies or unexpected speech output from these bots are sometimes being raised as part of the feedback for the test plans themselves.
However, there doesn't appear to be a clear, defined process for addressing this specific type of feedback. Could we clarify the intended procedure?
Specifically:
- Who is responsible for investigating potential issues with the bot's speech output?
- Should testers continue to report these speech output observations as part of the general test plan feedback, or should separate issues be filed specifically for bot speech output problems?
Having a defined and documented approach for handling these observations would be helpful for everyone involved.
Thank you!
The ARIA-AT Community Group just discussed Issue 1240 - Reporting feedback on bot output and performance.
The full IRC log of that discussion
<jugglinmike> Topic: Issue 1240 - Reporting feedback on bot output and performance<jugglinmike> github: https://github.com/w3c/aria-at/issues/1240
<jugglinmike> Matt_King: IsaDC raised this today
<jugglinmike> Matt_King: Joe was raising issues for VoiceOver bot problems
<jugglinmike> Matt_King: I changed the title on those three issues, from "Feedback:" to "VoiceOver Bot Feedback:"
<jugglinmike> Matt_King: I've done that a few times in the past as well; I don't know if we want to continue doing that moving forward
<jugglinmike> IsaDC: Those issues have an impact within our data
<jugglinmike> IsaDC: We haven't documented how to deal with them
<jugglinmike> Matt_King: When we get feedback on a test plan, they get linked to the test plan. And we have to close all the issues on a test plan in order to advance it in some circumstances
<jugglinmike> Matt_King: So this workflow creates issues that are tied to a test plan when they actually should be associated with a bot
<jugglinmike> Matt_King: I assume it's really helpful to have information about the specific test plan version and test when people are giving feedback on the bot
<jugglinmike> Carmen: It is. We use that information to replicate the issues
<jugglinmike> Carmen: Perhaps I can do something to move them?
<jugglinmike> Matt_King: You could edit the description of the issue and remove the HTML comments (which are hidden when the issue is rendered on the page)
<jugglinmike> Matt_King: If those get deleted, I don't know if it would delete the linking in real time, or if the link information is static
<jugglinmike> Carmen: I don't know either, but I can ask howard-e and get back to you
<jugglinmike> Matt_King: Sounds good. howard-e will know
<jugglinmike> Matt_King: This solution is a little hacky, but it's probably better than writing additional code
<jugglinmike> carmen: Hopefully we will soon reach a world where consistency issues are no more!
<jugglinmike> IsaDC: The other part of the concern was that, for example when we have an issue raised against the test plan itself (e.g. to fix the command), we respond to say that we're working on it. How does this work with the bots?
<jugglinmike> Carmen: I add it to the project, but I don't know if that would be visible to you. Would you like me to add a comment?
<jugglinmike> IsaDC: That would be lovely
Next steps:
- [ ] Bocoup to check on how the issues raised can be unlinked from the test plans on the app
- [ ] @ccanash from Bocoup will confirm on tasks that these have been received and will be addressed.
Hi @IsaDC , I have created a new task https://github.com/w3c/aria-at-app/issues/1399 in the app and we have started the discussion internally. We will come back with a proposal. Is it ok if we close this task?