Review/update NVDA browse mode arrow commands in all radio test plans

[mcking65]

Per discussion in May 7, 2025 meeting, we need to review use of NVDA up, down, and insert+up commands in browse mode when testing any of the radio group examples.

The CG suspects some of the output recorded for these commands when testing the radio group examples is not based on actual default behavior. In all cases, we believe that multiple radio buttons would be announced if default settings are being used.

We need to:

1. Re-run test plans and pay careful attention to output for these commands.
2. Determine if the commands should be removed from the tests. It could be that the commands are not designed to accomplish the task specified by the test. Related:

• #1242
• #1243

[css-meeting-bot]

The ARIA-AT Community Group just discussed Issue 1246 - Arrow keys in NVDA test plans.

The full IRC log of that discussion

<jugglinmike> Topic: Issue 1246 - Arrow keys in NVDA test plans
<jugglinmike> github: https://github.com/w3c/aria-at/issues/1246
<jugglinmike> Matt_King: I think the Bot is adding a bunch of things to the queue because maybe some of the results don't match...?
<jugglinmike> Isa: On my tab, I can't see any progress
<jugglinmike> Matt_King: Let's see. "Action Menu Button"... "NVDA"... The bot has 11 of 11 responses recorded
<jugglinmike> Matt_King: Wasn't it going to be the case that these would only show up in the manual test queue if the responses didn't match prior responses and there were conflicts?
<jugglinmike> ChrisCuellar: I think that was the intention. But we're seeing all of them in the queue, already, but it looks like verdicts have not yet been recorded
<jugglinmike> Isa: It appears as though you have to assign testers
<jugglinmike> Matt_King: We have a little bit of a mess, but we can work around it
<jugglinmike> Matt_King: This might be what Joe_Humbert was talking about
<jugglinmike> Matt_King: So "radio group example using active descendant" is the one that I wanted to talk about here
<jugglinmike> Matt_King: We were going to re-run this with NVDA, and we have Isa and Louis assigned.
<jugglinmike> Matt_King: But it doesn't look like you've started on this
<jugglinmike> Matt_King: The goal here was, in particular, to look at the output for "insert+up arrow" because we think the output for "insert + up arrow" should always be giving information for all the radio buttons if the defaults are truly being used
<jugglinmike> Matt_King: At least that's what James was asserting when we last discussed the issue
<jugglinmike> Matt_King: We wanted to get two people to verify whether that is the case. And then we wanted to decide whether to remove "insert+up"
<jugglinmike> Matt_King: When we looked at the reports for "Active descendant", it looked like the arrow keys were fine. But James said, "I don't think they should be passing"
<jugglinmike> Matt_King: That's why we wanted to re-run--to see if other people get the same thing as James
<jugglinmike> Matt_King: We decided to just do the "Active descendant" first
<jugglinmike> Matt_King: We have Isa and Loius assigned to this "Active descendant" one
<jugglinmike> Isa: I think we should wait for the bot to finish whatever it's doing because it's currently very confusing
<jugglinmike> Matt_King: Agreed
<jugglinmike> Isa: So we don't modify anything, yet, right?
<jugglinmike> Matt_King: Right
<jugglinmike> Matt_King: I found out that sometimes I thought I was using the defaults with NVDA, but it wasn't actually using the defaults
<jugglinmike> Matt_King: By the way, I guess we should be looking at the instructions in the test plans
<jugglinmike> Matt_King: So the next step here is for Isa and Louis.
<jugglinmike> Isa: It's confusing right now, but we can try to work around it. If not, we'll ask for help on the mailing list
<jugglinmike> Isa: It would be helpful if we could have the process documented to know what to expect
<jugglinmike> Isa: The process with the bot and the automatic reports
<jugglinmike> Matt_King: I think it's behaving unexpectedly, so we want to correct that
<jugglinmike> Matt_King: We expected it to automatically publish reports if the new version of the screen reader generate the same output as the prior version
<jugglinmike> Matt_King: If the output was different,then a human would need to step in and review
<jugglinmike> ChrisCuellar: That's my understanding, as well
<jugglinmike> ChrisCuellar: I'm currently communicating with howard-e to understand what's gone wrong

[css-meeting-bot]

The ARIA-AT Community Group just discussed Issue 1246 - Arrow keys in NVDA test plans.

The full IRC log of that discussion

<howard-e> TOPIC: Issue 1246 - Arrow keys in NVDA test plans
<howard-e> github: https://github.com/w3c/aria-at/issues/1246
<howard-e> Matt_King: I was looking into what was blocking the radio group test plan and it's this issue
<howard-e> Matt_King: We have all the reports done except for NVDA and they're conflicts. WHich are related to this issue
<howard-e> Matt_King: would like for us to prioritize getting this issue resolved and the first step is re-running these plans in the Test Queue
<howard-e> Matt_King: This is the Feb 12 Radiogroup using aria-activedescendant. Isa and LouisDo are assigned here and no work seems to be started there just yet
<howard-e> Matt_King: But looks like you have no bot results for yours Isa
<howard-e> Isa: I'll re-run the bot for myself then
<howard-e> Matt_King: Is this something you can prioritize this week LouisDo?
<howard-e> LouisDo: Yep
<howard-e> Matt_King: One thing particularly is if the bot gathered the right input for the up and down arrow commands
<howard-e> LouisDo: I can definitely pay attention to those
<howard-e> Matt_King: Looking right now at test 1's down arrow, down arrow command. Maybe we could move faster if .. Looking at the bot run assigned to Louis right now ...
<Matt_King> NVDA output from test 1 down arrow command:
<Matt_King> heading
<Matt_King> level 3
<Matt_King> Pizza Crust
<Matt_King> grouping
<Matt_King> radio button
<Matt_King> not checked
<Matt_King> Regular crust
<Matt_King> radio button
<Matt_King> not checked
<Matt_King> Deep dish
<Matt_King> radio button
<Matt_King> not checked
<Matt_King> Thin crust
<howard-e> Matt_King: So the bot recorded output for all of the radio buttons. So the bot is giving results for the same thing James was getting
<howard-e> Matt_King: So this is different if you look at the same results from the reports page
<howard-e> Matt_King: Actually, the down arrow, down arrow is the same here. It was the ins+up arrow there was a disagreement
<howard-e> Matt_King: Maybe it's the insert + Up arrow we were concerned with when raising the issue.
<howard-e> Matt_King: The test says "navigate forwards into a radio group where no radio buttons are checked" so not necessarily pinpointing any
<howard-e> Matt_King: So if we include down arrow for NVDA then the assertions would definitely be wrong
<howard-e> Matt_King: In test 7 "navigate forwards to unchecked radio button", we didn't include the arrow key in that one, we just did f and r and in test 8, shift + f and shift + r
<howard-e> Matt_King: same with tests 9 and 10
<howard-e> Matt_King: But in test 15, we have insert + up arrow and the report says ... but even the test name says we're collecting information about a radio button, and also 14
<howard-e> Matt_King: It says out of grouping
<howard-e> Isa: I think the bot was giving wrong results
<howard-e> LouisDo: From what I was testing, the commands I was testing gave right output but the bot was doing something different
<howard-e> LouisDo: I think the bot was executing up arrow but not insert + up arrow
<howard-e> LouisDo: so i should still prioritize in that way?
<howard-e> Matt_King: Wondering if we should even restructure the tests in that way based on that
<howard-e> Matt_King: So the decisions to make here is if we should remove down arrow from tests 2 and 3
<howard-e> jscholes: If you're in focus mode, and press insert + up, you should here only that radio button and not any of these others
<howard-e> Matt_King: The bot is also mentioning 'top' in the output which makes me think it might be on an edit field
<howard-e> jscholes: top only gets uttered if you move the cursor to the beginning of something it is already on
<howard-e> jscholes: it happens with desktop mode with numpad 7 and laptop layout when you do insert + up
<howard-e> jscholes: could the bot be configured to use laptop layout?
<howard-e> Matt_King: It seems like the bot is configured in that way
<howard-e> Isa: It is invalidating the commands in some way then, because it would mean the bot isn't following the default settings instructions
<howard-e> Matt_King: And in laptop layout, if you press insert + up arrow in browse mode, it would actually be ignoring the insert
<howard-e> jscholes: well in browse mode, when you move the arrow keys, you're moving the NVDA document cursor. If you press insert + up or numpad 7, you move the view cursor and not the browse cursor
<howard-e> jscholes: you would move the review cursor up by 1 line so it's impossible to say from the output alone which cursor it's moving
<howard-e> Matt_King: But it does appear to be consistent with laptop mode
<howard-e> jscholes: exactly
<howard-e> Matt_King: Well 2 things, 1 is we may have found a bot bug and 2 is making a decision on how to modify all 3 tests
<howard-e> Matt_King: Tests 14 and 15. Or any test when reading the current radio button, we want to read the name of radio button and it's state?
<howard-e> jscholes: I would find it odd not to also mention the role
<howard-e> Matt_King: So maybe we don't test ins + up arrow at all or we just test it for name, role and state
<howard-e> jscholes: But it would fail
<howard-e> Matt_King: right, but not all. would get the name
<howard-e> Matt_King: Do we think in browse mode, because the default is screen layout that it's legit for us to just not test insert + up arrow?
<howard-e> LouisDo: The objection to that is that combination is too verbose, no?
<howard-e> Matt_King: Well insert + up arrow according to NVAccess, is that it's not the intent so that's an argument in favor of leaving it out
<howard-e> jscholes: In a way, they are designed to provide the function but as a subset of what they do
<howard-e> Matt_King: it gives a lot of information but I suppose you just don't know which button is the current button
<howard-e> Matt_King: we have things like position but it doesn't tell you what you're on. But the whole purpose of the test is telling you about one specific button
<howard-e> Isa: In a way it does, but it gives more
<howard-e> LouisDo: Right and that's probably NVAccess position as well, but we aren't expecting it in that way
<howard-e> jscholes: It's like in a way, calling a "say all" and it said too much things. I don't think it's excess verbosity as it's behaving as intended though
<howard-e> Isa: So it's not necessarily failing so I don't think we should leave it out
<howard-e> jscholes: yes, i don't think we leave it out. But if we include these assertions, it will pass them. People tend to not really notice what's missing
<howard-e> LouisDo: So it's moot?
<howard-e> jscholes: NVAccess may have more of an issue with leaving them out
<howard-e> jscholes: I still don't think it makes sense to navigate into the end of a radio list with an up arrow because you can't do that
<howard-e> Isa: That may be a different test jscoles
<howard-e> Matt_King: One of things that give me concern now is in which of the reports, is that the output for insert + up arrow is inaccurate
<howard-e> Matt_King: We need to scrutinize the insert + up arrow in all the results
<howard-e> Matt_King: I also want to go back and take a fresh look at those conflicts
<howard-e> Matt_King: Also including insert + down and raise an issue for each result we find