Instability of finding elements of UWP Xamarin app on Windows 11

[VladislavShcherba]

On our project, test automation team has chosen to use WinAppDriver as a tool for automation of desktop applications. Our application under test is a UWP app build with Xamarin framework. Our tests are running on Windows 10 and Windows 11 VMs hosted by Microsoft in Azure pipelines. We use Java language binding, WinAppDriver v1.2.1.

Although we were able to achieve a stable work of test automation framework on Windows 10 machines, we are still having troubles with Windows 11 machines.

At first, tests runs on Windows 11 were very unstable. More than in 50% of cases tests were failing right after the driver initialization on the first find operation of the username element on login page. Tests runs on Windows 10 didn't have this problem. What helped to solve the problem is a workaround to launch the application under test without the usage of the driver, but by some other means (we use 'microsoft.windows.softwarelogo.appxlauncher.exe'), and then to initialize the driver by found main window handle.

But as the number of tests grows, it becomes visible that the problem didn't go away completely. Rarely, but still the problem is visible that some random element on the screen can't be found. This element was found a lot of times before in other tests executions. And it will be found a lot of times in the future executions. It does not depend on the 'difficulty' of the element to find. We have seen that behavior with the elements that are found just by unique AutomationId on the page.

The test automation framework is written in a way that all find operations use explicit waits inside. Also we are making video recordings of each test execution. So it is possible to review what was going on with the test. And it is evident that the element does exist on the screen. And the framework tries to find it on the appropriate screen, within the appropriate timeout, before it fails the test.

Could you please help us with some suggestions on how to solve the problem? Does WinAppDriver really supports Windows 11 well on practice from your experience? I saw that the official answer is 'yes', but I also see that the last version of WinAppDriver was released in 2020, before Windows 11 released (2021).

Thanks in advance!

[anunay1]

Can you share some piece of code and also what exception you get when the execution fails?

[Shakevg]

I suppose it is just related to some different behavior under Win 11 (I saw that it can be different behavior and controls under different Win build versions). What timeout are you using in explicit wait? Did you try to save and check PageSource on Fail? Did not find an element exist in PageSource? Did you search element under root session or some parent element?

[VladislavShcherba]

@anunay1 Thanks for responding! I'm receiving org.openqa.selenium.NoSuchElementException after the framework tries to explicitly find an element within 15 seconds. WinAppDriver logs show the attempts:

==========================================
POST /session/3F1E2A88-77DC-432B-A99C-DCE9DE3C8447/element HTTP/1.1

Accept-Encoding: gzip

Connection: Keep-Alive

Content-Length: 63

Content-Type: application/json; charset=utf-8

Host: 127.0.0.1:4723

User-Agent: selenium/3.141.59 (java windows)



{
  "using": "accessibility id",
  "value": "id_goes_here"
}
HTTP/1.1 404 Not Found

Content-Length: 139

Content-Type: application/json



{"status":7,"value":{"error":"no such element","message":"An element could not be located on the page using the given search parameters."}}


==========================================
POST /session/3F1E2A88-77DC-432B-A99C-DCE9DE3C8447/element HTTP/1.1

Accept-Encoding: gzip

Connection: Keep-Alive

Content-Length: 63

Content-Type: application/json; charset=utf-8

Host: 127.0.0.1:4723

User-Agent: selenium/3.141.59 (java windows)



{
  "using": "accessibility id",
  "value": "id_goes_here"
}
HTTP/1.1 404 Not Found

Content-Length: 139

Content-Type: application/json



{"status":7,"value":{"error":"no such element","message":"An element could not be located on the page using the given search parameters."}}


==========================================
POST /session/3F1E2A88-77DC-432B-A99C-DCE9DE3C8447/element HTTP/1.1

Accept-Encoding: gzip

Connection: Keep-Alive

Content-Length: 63

Content-Type: application/json; charset=utf-8

Host: 127.0.0.1:4723

User-Agent: selenium/3.141.59 (java windows)



{
  "using": "accessibility id",
  "value": "id_goes_here"
}
HTTP/1.1 404 Not Found

Content-Length: 139

Content-Type: application/json



{"status":7,"value":{"error":"no such element","message":"An element could not be located on the page using the given search parameters."}}

...

I am under NDA, so I have changed the real ID to "id_goes_here".

I can only show some of the standard little bits of code that are more or less common in automation projects. But then sharing it won't help, right?

[VladislavShcherba]

@Shakevg Thanks for responding!

The timeout is 15 seconds and it is more than enough for the page to load.

For the page source, I will let you know as soon as I figure that out.

Currently the application under test is launched by the help of microsoft.windows.softwarelogo.appxlauncher.exe. Then the driver is initialized with the Root session to find the window handle of the application under test. Then the driver is initialized using appTopLevelWindow capability. All tests use this driver. So the elements of the application under test are found by the driver initialized specially for that application. Do you mean using the driver initialized with the Root session for the whole automation framework can be a workaround here?

Sometimes the issue is visible even with the easiest elements that are found by AutomationId starting from the driver itself, without parent elements.

[anunay1]

@anunay1 Thanks for responding! I'm receiving org.openqa.selenium.NoSuchElementException after the framework tries to explicitly find an element within 15 seconds. WinAppDriver logs show the attempts:

==========================================
POST /session/3F1E2A88-77DC-432B-A99C-DCE9DE3C8447/element HTTP/1.1

Accept-Encoding: gzip

Connection: Keep-Alive

Content-Length: 63

Content-Type: application/json; charset=utf-8

Host: 127.0.0.1:4723

User-Agent: selenium/3.141.59 (java windows)



{
  "using": "accessibility id",
  "value": "id_goes_here"
}
HTTP/1.1 404 Not Found

Content-Length: 139

Content-Type: application/json



{"status":7,"value":{"error":"no such element","message":"An element could not be located on the page using the given search parameters."}}


==========================================
POST /session/3F1E2A88-77DC-432B-A99C-DCE9DE3C8447/element HTTP/1.1

Accept-Encoding: gzip

Connection: Keep-Alive

Content-Length: 63

Content-Type: application/json; charset=utf-8

Host: 127.0.0.1:4723

User-Agent: selenium/3.141.59 (java windows)



{
  "using": "accessibility id",
  "value": "id_goes_here"
}
HTTP/1.1 404 Not Found

Content-Length: 139

Content-Type: application/json



{"status":7,"value":{"error":"no such element","message":"An element could not be located on the page using the given search parameters."}}


==========================================
POST /session/3F1E2A88-77DC-432B-A99C-DCE9DE3C8447/element HTTP/1.1

Accept-Encoding: gzip

Connection: Keep-Alive

Content-Length: 63

Content-Type: application/json; charset=utf-8

Host: 127.0.0.1:4723

User-Agent: selenium/3.141.59 (java windows)



{
  "using": "accessibility id",
  "value": "id_goes_here"
}
HTTP/1.1 404 Not Found

Content-Length: 139

Content-Type: application/json



{"status":7,"value":{"error":"no such element","message":"An element could not be located on the page using the given search parameters."}}

...

I am under NDA, so I have changed the real ID to "id_goes_here".

I can only show some of the standard little bits of code that are more or less common in automation projects. But then sharing it won't help, right?

Can you check the page source when in fails?

[VladislavShcherba]

@Shakevg @anunay1 I have added logging and now the framework prints a page source on a test fail. And I see that in the case when an element was not found without any good reasons, resulting page source is 'corrupted'. By 'corrupted' I mean that it does not have some elements that are obviously present on the screen (including the one that the framework was trying to find). So this confirms my initial guess that the instability of the test executions comes from WinAppDriver itself, not from our framework.

I must admit that we are seeing similar behavior with inspect.exe tool. When using our application under test with launched inspect.exe, after some time inspect.exe starts behaving inappropriately. The chance of such behavior increases the more time you have spent and the more 'clicks' you have done in the AUT. The inspect.exe starts to show erroneous information and/or do not show all elements. The only way to workaround I've found is to close the AUT and inspect.exe, then open the AUT, go until desirable screen, open inspect.exe. It is inconvenient and slow, but at least it does not block us. Unlike the WinAppDriver problem, which makes the automation not so useful, since there is always a chance (even though not so high) that the test fails due to driver issue. But it seems to me that this two problems are related to each other...

Do you know if there is some workaround for this? What can I try to do to completely get rid of this problem? Thanks!

[anunay1]

Are you using a lot of xpaths in your framework?

[VladislavShcherba]

@anunay1 There are a few xpaths in tricky places, but the most of the locators are AutomationId.

[Shakevg]

VladislavShcherba Can you try to create a new session for each test by process name and check if it will help?

[VladislavShcherba]

@Shakevg Thanks for answering! This was done almost as soon as the automation project has started. And I think it helps to improve stability. I have a feeling that the more requests WinAppDriver process has handled, the more chances that it will have problems on finding elements.

We have 'short' and 'long' tests. I know it is always better to have 'short' UI tests, but we also wanted to check a walk through the whole workflow of the app since it is very sequential. According to my observations we have more problems with the 'long' tests. It may be not only because these tests do more actions, so more potential places to fail, but also because WinAppDriver process is the same for the whole test, and it seems that WinAppDriver has kind of 'accumulation effect'. Have you noticed that effect based on your experience?

I think the solution for us is to re-execute the test that failed with NoSuchElementException on Windows 11. Also to restart WinAppDriver (and it might not help without restarting AUT either, so also to restart AUT) right during the execution of a long tests.