accessibility-insights-windows
accessibility-insights-windows copied to clipboard
microsoft
[Feature Request] Add Ability to Expand Nodes from UI Inspection Tree
Is your feature request related to a problem? Please describe. At the moment, if you have a UI element under inspection there is no way to dive deeper into its children nodes. You can get observe and navigate its top-level children, but nothing deeper.
Describe the solution you'd like It would be nice to be able to dive deeper into children UI elements from the inspection window.
Describe alternatives you've considered It is possible to inspect UI elements deeper in the tree on screen, but it will not let the user easily observe elements that may perhaps be hidden from the UI or collapsed.
This issue has been marked as ready for team triage; we will triage it in our weekly review and update the issue. Thank you for contributing to Accessibility Insights!
@adrianaandreeva this request has been heard from the substantial number of users. could we create a feature for this in the next quarter?
This issue has been marked as requiring additional investigation by the Accessibility Insights team. When the issue is ready to be triaged again, please update the issue with the investigation result and add "status: ready for triage". Thank you for contributing to Accessibility Insights!
Current thinking: Add 2 options to the drop-down menu on the element. One allows one generation of children to the enumerated for this element, the other expands all descendants of the tree. We'll want to get input from @cheade on the specifics
Simplest thing is to have a button to walk the whole tree. Let's connect with @cheade to get some inputs here.
@Sirisha-Kommushetty Another option would be to add a "Show Descendants" menu item alongside "Show Ancestry":
@enricotj - We've looked into this and are exploring possible solutions.
@jalkire What's the difference between 'Show ancestry' and 'Show descendants'?
Awesome! Thanks for all your great work! ^_^
On Mon, Aug 5, 2019, 11:53 AM Chelsea Adelman [email protected] wrote:
@jalkire https://github.com/jalkire What's the difference between 'Show ancestry' and 'Show descendants'?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/accessibility-insights-windows/issues/333?email_source=notifications&email_token=ABOFWR6CKJOWN4J44W5ZPB3QDBZJDA5CNFSM4HH6KT3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3SXQDQ#issuecomment-518354958, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOFWR4Y4AZBMCV4Z5JRXLTQDBZJDANCNFSM4HH6KT3A .
+1
I frequently find myself falling back to the legacy Inspect app just because I find it so much easier to navigate around the UIA tree.
I'd urge you against scoping this feature to just descendants. What if I want to see siblings? What if I want to see descendants of siblings?
@cheade "Show ancestry" toggles displaying the hierarchy tree from the selected element all the way back to the root/desktop node:
Checked:
Unchecked:
In some ways, it is analogous to this feature request, except it allows inspecting higher up the tree rather than descending into the tree.
On way to offer the requested functionality would be to add a menu item like "Show all descendants" which, if checked, would result in ai-win fully populating the descendants of the selected element in the hierarchy tree.
Can we replace "Show descendants" with "Show all descendants?" What's the benefit of having the two options?
We don't currently have either--I didn't mean to suggest adding two new items. "Show descendants" or "Show all descendants" are two options for the text of the single new menu item I am suggesting.
@cheade - We have some ideas on what we could do here. Can we run by UX interviews for the next steps.
+1 from me. Also, I think it would be nice to extend this feature even further: It should be possible to visualize the whole UI automation tree similar to how it is visualized in inspect.exe. This way, one could navigate to any element in any running application by clicking it in the UI tree instead of clicking the element in the UI.
This issue has been marked as ready for team triage; we will triage it in our weekly review and update the issue. Thank you for contributing to Accessibility Insights!
+1 as a useful feature that would help me transition fully off of Inspect. A specific case that's interesting is when a list contains child items that a dissimilar to each other. Being able to see all list items simultaneously would help highlight the differences between them.
I like the idea of a "Show Descendants" toggle alongside the "Show Ancestry" toggle. Perhaps in that case the latter could be renamed to "Show Ancestors" for more symmetry. Functionally, I could see a few options for it.
- Show all descendants for just the leaf node(s).
- Show all descendants for the currently selected node.
- Show all descendants for all currently visible nodes. If ancestors are also shown, this would display the whole tree. Otherwise, it would display the whole subtree for your currently selected node.
1 or 2 would be good for performance since there are fewer elements to query, but it doesn't deliver on all the use cases for seeing all elements. 3 is great for completeness, and so that you can click around and inspect different nodes in the tree without changing what is displayed in the tree. 3 would get my vote.
I think this feature would also add a lot of value to the "Entire app" selection mode. In that mode, I think the typical expectation is to see the complete tree (as in option 3), whereas currently you see just the root node and one level deeper.
This lead me to some false conclusions when trying to diagnose an issue my team faced. Because I couldn't hover an element, and because the UI did not indicate the parent element had children, I falsely concluded that the element was empty.
Using Ctrl+Shift+F7 and Ctrl+Shift+F8 "worked around" this issue, but this is a major blocker for me to replace Inspect.exe.
It's bothered me for years, but this time it actually caused me to misdiagnose something. Happy to give more help as needed :)!
EDIT: adding concrete requests
My feature priorities
For me:
- P0: visual indication that an item has children (an unexpanded caret, for example). Otherwise, it's easy to miss items in the tree.
- P0: app-based way to enumerate an item's children (clicking the caret, or pressing right arrow key when the tree item is selected). Ctrl+Shift+F7 is buried and hard to press.
- P1: show the current app's entire visual tree, always (instead of strictly ancestors). I often want to walk through my apps' accessibility trees holistically, to make sure there are no surprising elements. To do so, I need to view the entire UIA tree.
- P2: let me show the entire visual tree for all of Windows (like Inspect.exe). This is pleasant when doing larger debugging across multiple windows (and when working with tooltips and random tool windows), but it is not an everyday requirement for me. I'd certainly use it if it existed (I might even set it as my default).
I can imagine needing a way to refresh an item's children, too. Collapsing and re-expanding an item does that inspect.exe I think. I'd be happy with that or a refresh button next to the item in the tree.
Which descendants to show
Thinking about the proposals @alexhinz made (and some earlier thoughts in the thread):
- Show all descendants for just the leaf node(s).
- Show all descendants for the currently selected node.
- Show all descendants for all currently visible nodes. If ancestors are also shown, this would display the whole tree. Otherwise, it would display the whole subtree for your currently selected node.
(Plus, say, 4. Show all descendants for all nodes across all of Windows)
I have no practical purpose for 1 or 2; if I had to choose, I'd always go with "show everything for my entire window" (3ish) or "show everything for all of Windows" (4) and just to move around in the tree view of UIA elements.
I've seen a couple references to performance, but I'd ask the Accessibility Insights team to deep-dive into that—is there a practical perf benefit to only enumerating descendants for the selected note? inspect.exe seems to be fine listing everything in Windows whenever I ask, so I'd be surprised if there's a large concrete perf impact to enumerating everything.
I'll put it another way: if I woke up tomorrow and AI showed my entire window with all items in the tree all the time, I'd be delighted. I don't want to encourage the team to be overzealous, but I'd like to be sure there's a clear reason to keep the "only ancestors" or add an "only ancestors & descendants" mode if the team chooses to do so.
This issue requires additional investigation by the Accessibility Insights team. When the issue is ready to be triaged again, we will update the issue with the investigation result and add "status: ready for triage". Thank you for contributing to Accessibility Insights!
We want to pursue this. Let's try to get a 1-engineer spike to understand the options, including potentially using the updated UIA mechanisms
If anything can be done to improve navigating the UIA tree with keyboard arrow keys, I'd encourage it. This involves exploring descendants too. Inspecting a tree and having to use triple chorded keystrokes (e.g., control + shift +f*) is inefficient and ineffective. And I support @citelao's idea for showing the whole tree wholistically, like in Inspect.
Unfortunately, this is out of scope given our current priorities.
Thanks for using Accessibility Insights!
@DaveTryon Is this something you'd be open to contributions for?
This is a long-standing regression from inspect.exe, and makes it very difficult for people who work in accessibility, like me, to audit UI to make sure the whole tree is correct.
I have a hunch the initial design was based on Edge's behavior, but Edge is actively changing their accessibility UI to show the complete tree:
Complete tree: