Improve hint enumeration performance

[zsims]

Enumerating hints for "busy" windows can be quite slow, sometime taking ~3 seconds.

[zsims]

Can confirm that this is mostly in the way AutomationElement.FindAll is being used. It supports "caching" but only after the first visit :( https://msdn.microsoft.com/en-us/library/system.windows.automation.automationelement.findall%28v=vs.110%29.aspx

[oysteinkrog]

I've been playing with it a bit today but was unable to improve performance much, things I tried:

1. Switch to COM-based UI automation API (http://uiacomwrapper.codeplex.com). I think this gave a (very) slight performance improvement. It seems that the .net wrapper can be a bit slow for some cases.
2. Enumerate only certain control types (e.g. buttons), no real gain.
3. Enumerate ahead of time (when active window changes). I have not yet been able to get this to work as the UI automation API throws a TypeLoadException, will continue working on this as it seems the most promising.

[zsims]

Thanks a lot for contributing :smile:. I've also (albeit briefly) thought about your third option, does introduce some interesting challenges, the idea being to cache top level elements on focused window change? Should work in theory :)

Feel free to assign this issue to yourself.

[zsims]

I tried another option, using the in-built cache functionality of the automation framework. With negative results :disappointed:

For reference, 3x runs (in a row without re-launching), timing how long it took to create a hint session in UiAutomationHintProviderService.cs

Cache (With current API)

See 09f4aa86350b090d03e467b87b9a46548bd3bda4

Wordpad Before = 884 + 604 + 752 ms After = 1092 + 996 + 945 ms

Word Before = 519 + 604 + 547 ms After = 755 + 677 + 701 ms

COM API Wrapper

Per https://uiacomwrapper.codeplex.com/, See be48062f99d1ca796c6698e034c968d268043b46

Wordpad Before = 884 + 604 + 752 ms After = 970 + 911 + 1100 ms

Word Before = 519 + 604 + 547 ms After = 1281 + 1132 + 1101 ms Will try your ahead-of-time idea when I can :+1:

[zsims]

More playing :dancers:. I whipped up your idea -- basically, build the HintSession ahead of time to mitigate the performance problems with the UIA API, see ec29786ebc7407d8ef6a83c4ecde7e6581d2eee2. It works quite well, assuming the hint session has already been built.

It's still WIP, but:

1. Uses Windows Events to detect when a new window becomes focused
2. Doesn't yet handle Windows with dynamic content that well
3. Doesn't handle window resize/scroll
4. Sometimes, it's not "ahead" enough. E.g. using Explorer, hints can take ~500ms to enumerate but I already know what I want to activate.

Cheers for the idea :smile:

[oysteinkrog]

Ah cool!, I see that you invoke on the gui thread, that must be the fix for the exception I was struggling with! (I should have known:P) I'm not sure if you looked at any of the wip code I have in my fork... I did pretty much what you did: https://github.com/oysteinkrog/hunt-and-peck/commit/48d4da2712cb80466d8f533c4b9c0a90d9a96118 I decided to also replace the windows forms stuff with a low-level native window: https://github.com/oysteinkrog/hunt-and-peck/commit/7965106fe749d781a81781b18a62aa4238ece6d5 I also played around with AccessKeys, I'm considering adding another mode/hotkey that shows the access keys instead of random keys, would be a good mode for learning to use them.

[zsims]

I see that you invoke on the gui thread, that must be the fix for the exception I was struggling with! (I should have known:P)

Seemed to help, I'm not 100% sure if it's required. Bit rusty with COM.

I'm not sure if you looked at any of the wip code I have in my fork...

Only just looked. Your stuff is much cleaner :+1:

Agree re: AccessKeys, even just making sure the same label is applied to the same hint (or there abouts). It's a bit frustrating that the label changes each time for the same button.

I think the ahead-of-time hint session is a good idea, and seems to work, but not sure how to handle the case of windows that change -- short of digging into a lot more of the events. Any ideas here?

[zsims]

Ah cool!, I see that you invoke on the gui thread, that must be the fix for the exception I was struggling with! (I should have known:P)

Hmm, in hind sight... I think this is wrong according to MSDN

Because of the way Microsoft UI Automation uses Windows messages, conflicts can occur when a client application attempts to interact with its own UI on the UI thread.

There is a brief Window when the app launches (although should be filtered by WINEVENT_SKIPOWNPROCESS), and I suspect this was causing issues but the Dispatcher.Invoke essentially "defers" this. Just a theory, but probably needs to be "on another thread" all together. Task.Run(() => { ... }) also works :/

I've also been looking at the UI automation events, wonder if this is a good way to detect "changes" in Windows?

[zsims]

For reference, pretty good blog on improving UIA performance: https://blogs.msdn.microsoft.com/winuiautomation/2013/06/04/a-little-uia-qa/ (TL;DR: caching and the native COM API)

[zsims]

Switched to the native API, this has made a drastic improvement.

In Visual Studio (start page, average across 3 clean runs): Before: ~2024ms After: ~616ms

Still room for improvement as this isn't using any of the UIA caching APIs.

[extratype]

I discovered enumeration takes much longer if ran on kind of.. background (tested on Windows 10). There's no official way to make it faster. I found a dirty hack (https://github.com/extratype/hunt-and-peck/commit/55c4377b4fda458c7c20c09480df9c323c53188b) but it's not always working.

[zsims]

Interesting... Do you know if there's any documentation in what impact this has? Didn't seem to make much of a difference for me. And https://msdn.microsoft.com/en-us/library/windows/desktop/ms685100(v=vs.85).aspx doesn't really mention anything about priority and CPU load (which I assume is the goal?)

[extratype]

I couldn't find any. I have searched MSDN and this forum (https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/home?forum=windowsaccessibilityandautomation). I just figured it out that UI Automation uses WM_GETOBJECT internally and there's no message priorities we can control. I tried adjusting process/thread priorities, but no gain.

I merged the commit https://github.com/extratype/hunt-and-peck/commit/55c4377b4fda458c7c20c09480df9c323c53188b into the current master branch. Without this, enumerations for Windows explorer occasionally took ~100ms more. To reproduce, repeatedly press the hotkey with a short delay, and then with long delays. You will notice performance varies.

This matters in my branch (in hundreds of milliseconds), where I exploit UIA APIs to get only on-screen elements from an arbitrarily long list or tree view. It's much slower on average, but with the commit, it's barely tolerable.

[zsims]

Thanks for the investigation! It would be nice to get a consistent gain, even by enumerating hints ahead of time or changing the way the tree is walked. That being said, the switch to the native API has drastically improved things from what I've seen. Thoughts?

[extratype]

I agree on the ahead-of-time idea. You may try these testing tools included in Windows Kits to inspect elements and watch events to find something useful.