CommunityToolkit
RichSuggestBox Intermittent Test Failures
Describe the bug
At least some part of #347 seem to be intermittent failures in the RichSuggestBox tests as well
Here:
https://github.com/CommunityToolkit/Windows/blob/2c0f9b3d8096fce836cd239a9c5cad3494e236f7/components/RichSuggestBox/tests/Test_RichSuggestBox.cs#L143
And here:
https://github.com/CommunityToolkit/Windows/blob/2c0f9b3d8096fce836cd239a9c5cad3494e236f7/components/RichSuggestBox/tests/Test_RichSuggestBox.cs#L308
Both involving adding new tokens.
This seems to more often happen for WinUI 3, so possibly has to do with different behavior or regression there? But something definitely seems to be wonky and unstable here with this control and these tests.
Failed Test_RichSuggestBox_DeleteTokens (@Token1,#Token2) [108 ms]
Error Message:
Assert.AreEqual failed. Expected:<2>. Actual:<1>. Unexpected token count after adding.
Stack Trace:
at RichSuggestBoxExperiment.Tests.Test_RichSuggestBox.<>c__DisplayClass2_0.<<Test_RichSuggestBox_DeleteTokens>b__0>d.MoveNext() in D:\a\Windows\Windows\components\RichSuggestBox\tests\Test_RichSuggestBox.cs:line 143
--- End of stack trace from previous location ---
at CommunityToolkit.WinUI.DispatcherQueueExtensions.<>c__DisplayClass3_0.<<EnqueueAsync>b__1>d.MoveNext() in /_/components/Extensions/src/Dispatcher/DispatcherQueueExtensions.cs:line 180
--- End of stack trace from previous location ---
at RichSuggestBoxExperiment.Tests.Test_RichSuggestBox.Test_RichSuggestBox_DeleteTokens(String token1, String token2) in D:\a\Windows\Windows\components\RichSuggestBox\tests\Test_RichSuggestBox.cs:line 133
at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.ThreadOperations.ExecuteWithAbortSafety(Action action)
It's unclear if this is some underlying issue with the platform, or some race condition with us not waiting long enough in the test. But it needs investigation so we can have a more stable CI setup.
Steps to reproduce
Intermittent test failure
Expected behavior
Tokens always added
Screenshots
No response
Code Platform
- [ ] UWP
- [x] WinAppSDK / WinUI 3
- [ ] Web Assembly (WASM)
- [ ] Android
- [ ] iOS
- [ ] MacOS
- [ ] Linux / GTK
Windows Build Number
- [ ] Windows 10 1809 (Build 17763)
- [ ] Windows 10 1903 (Build 18362)
- [ ] Windows 10 1909 (Build 18363)
- [ ] Windows 10 2004 (Build 19041)
- [ ] Windows 10 20H2 (Build 19042)
- [ ] Windows 10 21H1 (Build 19043)
- [ ] Windows 10 21H2 (Build 19044)
- [ ] Windows 10 22H2 (Build 19045)
- [ ] Windows 11 21H2 (Build 22000)
- [ ] Other (specify)
Other Windows Build number
No response
App minimum and target SDK version
- [ ] Windows 10, version 1809 (Build 17763)
- [ ] Windows 10, version 1903 (Build 18362)
- [ ] Windows 10, version 1909 (Build 18363)
- [ ] Windows 10, version 2004 (Build 19041)
- [ ] Windows 10, version 2104 (Build 20348)
- [ ] Windows 11, version 22H2 (Build 22000)
- [ ] Other (specify)
Other SDK version
No response
Visual Studio Version
No response
Visual Studio Build Number
No response
Device form factor
No response
Additional context
No response
Help us help you
Yes, but only if others can assist.
Before these asserts we should wait for the UI to update, like we do in other tests:
https://github.com/CommunityToolkit/Windows/blob/605cd4369d0f46a0935af644282854798d6f317e/components/Primitives/tests/SwitchPresenter/SwitchPresenterTests.cs#L40-L41
See if that helps.
There's probably a few places in these tests we can do this and remove the Delay.
We should disable these tests with a preprocessor directive on WinUI 3.
I suspect since the UWP tests here always seem to run fine and fail on WinUI 3 that there's some new issue in the platform in this space.
It'd be great if we could isolate that on the RichEditBox somehow and file a platform issue.
I don't know if this is just a test issue or has implications on usage on the control for RichSuggestBox, I don't think we've had other reports. It's hard to know the impact of this. FYI @zadjii-msft [P2], as I believe you were tinkering with this lately, not sure if you noticed anything (and sorry for your loss).
