[Epic] Flaky tests roundup

[MartinZikmund]

Current behavior

This epic tracks flaky tests so they can be addressed one by one.

When a flaky test is added in this list, it can be set to [Ignore] on the platform or platforms where it tends to fail.

• When_Rectangle UI test is very flaky on iOS
• BreadcrumbBar leak tests are flaky on iOS
• Given_AppWindow.WhenResize is failing in CI on Linux, passing locally
• [ ] #17101
• [ ] #17102
• [ ] When_Image_Is_Monochromatic fails on iOS .NET 9 RC1
• [ ] https://github.com/unoplatform/uno/issues/18064
• [ ] Some Given_ImageBrush.WhenStretch fail on iOS
• [ ] When_Hungarian_Culture_Column_Order, When_Czech_Culture_Column_Order and When_CanadaFrench_Culture_Column_Order fail on net9 wasm. It could be related to a missing/filtered ICU, or a headless browser issue, it works locally.
• [ ] MoveItemsBetweenPrimaryAndSecondaryCommands is unstable on android
• [ ] When_DownSampling is flaky and disabled in https://github.com/unoplatform/uno/pull/16991

  Running test When_DownSampling()
  GLIB UNHANDLED EXCEPTIONSystem.ArgumentNullException: Value cannot be null. (Parameter 'bitmap')
     at SkiaSharp.SKImage.FromBitmap(SKBitmap bitmap)
     at Windows.UI.Composition.CompositionSurfaceBrush.UpdatePaint(SKPaint fillPaint, SKRect bounds) in C:\a\1\s\src\Uno.UI.Composition\Composition\CompositionSurfaceBrush.skia.cs:line 91
     at Windows.UI.Composition.CompositionSpriteShape.Paint(PaintingSession& session) in C:\a\1\s\src\Uno.UI.Composition\Composition\CompositionSpriteShape.skia.cs:line 19
     at Windows.UI.Composition.BorderVisual.Paint(PaintingSession& session) in C:\a\1\s\src\Uno.UI.Composition\Composition\BorderVisual.skia.cs:line 127
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 190
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
     at Windows.UI.Composition.Visual.RenderRootVisual(SKSurface surface, Nullable`1 offsetOverride) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 152
     at Uno.UI.Runtime.Skia.Gtk.SoftwareRenderSurface.OnDrawn(Context cr) in C:\a\1\s\src\Uno.UI.Runtime.Skia.Gtk\Rendering\SoftwareRenderSurface.cs:line 99
     at Gtk.Widget.Drawn_cb(IntPtr inst, IntPtr cr)
  

• [ ] #16167
• [ ] https://github.com/unoplatform/uno/issues/18105
• [ ] When_UnloadAndReload_Then_ReMaterializeItems
• [ ] When_ThemeChange on Wasm
  
• [ ] Android TestProgressRing_InitialState fails
• [ ] FlyoutTest_Unloaded
• [ ] FlyoutTest_Target is flaky on iOS
• [ ] https://github.com/unoplatform/uno/issues/13417
• [ ] PasswordBox_RevealInScrollViewer is flaky on iOS
• [ ] #15734
• [ ] TextBox_VerticalAlignment is flaky on Wasm
• [ ] Verify_MenuBarItem_Bounds_Native_Popups is flaky on Android - started being flaky after pull request #13630 (@ramezgerges)
• [ ] TextBox_Readonly is flaky on iOS
• [ ] When_Drag_Then_DoesNotScroll is not working on Android, the thumb drags when it should not
• [ ] CanChangeDisplayModeBeforeLoaded is flaky on Skia
• [ ] Holding_Tests.When_InListViewWithoutItemClick fails for missing item on android
• [ ] Holding_Tests.When_Basic fails on android
• [ ] Holding_Tests.When_InScroll fails on android
• [ ] Holding_Tests.When_InListViewWithItemClick fails on android
• [ ] TestTextBoxReleaseOutFocused is flaky on WASM
• [ ] https://github.com/unoplatform/uno/issues/15829
• [ ] When_MultipleItems is not failing sometimes (Timing related ?)
• [ ] Flyout_TemplatedParent is flaky on WASM
• [ ] https://github.com/unoplatform/uno/issues/15828
• [ ] https://github.com/unoplatform/uno/issues/15471
• [ ] TestTranslatedTap is flaky on WASM (could be related to https://github.com/unoplatform/uno/issues/13844)
• [ ] ValidateOverflowPlacement does not pass on Android
• [ ] TestTranslatedTap is failing on android
• [ ] DoesNotShowMenuIfSecondaryElementsAreCollapsed does not pass on Android
• [ ] When_NestedInSVAndOutOfViewportOnInitialLoad_Then_MaterializedEvenWhenScrollingOnMinorAxis is flaky on WASM
• [ ] #13844
• [ ] Android When_NestedIRSlowlyChangeViewport_Then_MaterializedNeededItems (Timeout even with retries) (Youssef: Can't repro)
• [ ] Android When_Verify_Canvas_With_Outer_Clip (Assert.Fail failed. ImageAssert.HasColorAt @ line 73) (Youssef: Verified - it always fails)
• [ ] UITests.Windows_UI_Xaml_Media.GradientBrushTests.GradientsPage crashes the android emulator completely (not just the app)
• [ ] Android VerifyNavigationViewItemToolTipPaneDisplayMode (Youssef: Verified - it always crashes locally)
• [ ] When_Focused_Element_Transformed is crashing the iOS/Android emulator
• [ ] Android VerifyCollectionChangeGetsRespected (Cleanup System.Exception: Event was not raised. in MUXApiTestBase.Cleanup()) (Youssef: test not disabled on master, and always passes for me)
• [ ] ListView_ObservableCollection_Unused_Space fails regularly on iOS (Expected: 132.0f But was: 88.0f)
• [ ] When_Open_Close_Root_Twice_Keep_State fails on android for missing TreeView items (Youssef: fails locally with null ref)
• [ ] When_UsingMultipleLanguages unit test fails when running with net7 or later (DateFormatter is buggy and CultureInfo itself returns wrong values)
• [ ] https://github.com/unoplatform/uno/issues/11997
• [ ] https://github.com/unoplatform/uno/issues/13257
• [ ] When_Grid_RowCollection_Changes fails with invalid child.SizePassedToArrangeOverride
• iOS 16+ tests
  • [ ] NativeCommandBar_Size fails (The first orientation change fails to be applied)
  • [ ] ListView_SelectedItems fails (Timeout ?)
  • [ ] When_Flipview_DataTemplateSelector (AssertFailedException: Assert.Fail failed. ImageAssert.HasColorAt @ line 101)
  • [ ] When_Add_Remove failure: ColorPicker
  • [ ] When_Add_Remove failure: CalendarView
• [ ] iOS When_ItemsControl.When_NestedItemsControl_RecycleTemplate (Assert.AreEqual failed. Expected a difference no greater than <1> between expected value <50> and actual value <0>. )
• [ ] iOS RepeaterTests.VerifyCorrectionsInNonScrollableDirection
• [ ] iOS/Skia VerifyFocusedItemIsRecycledOnCollectionReset (Skia fixed in https://github.com/unoplatform/uno/pull/13074)
• [ ] iOS When_WebMessageReceived (Timed out waiting for condition to be met. When_WebMessageReceived():207)
• [ ] iOS When_Flipview_Items_Modified (Timed out waiting for equality condition to be met. Expected -1 but last received value was 0.)
• [ ] iOS When_Render_Border_GetPixelsAsync (Assert.AreEqual failed. Expected:<<10, 10>>. Actual:<<10.5, 10>>. Invalid Layouted.)
• [ ] https://github.com/unoplatform/uno/issues/13095
• [ ] [All platforms] ValidateFractionalWidthDoesNotCrash (Assert.Fail failed. Timeout waiting on SpectrumRectangle.Fill to be set.) (Youssef: Test is actually skipped in WinUI. There is not much info on why it's skipped there, but I don't see anything that should trigger property changed for Fill)
• [ ] When_StretchAndAlignment on Android 28 (Youssef: Passing for me locally)
• [ ] TextBox_Readonly_ShouldNotBringUpKeyboard on Android 30 (The keyboard does not show up on emulators, breaking the test)
• [ ] When_PressOnContainerAndReleaseOnNested_Mouse on WASM (fails in CI)
• [ ] When_RepeatDuration: It was enabled in https://github.com/unoplatform/uno/pull/13060, but then failures are found and reverted in https://github.com/unoplatform/uno/pull/13089
• [ ] https://github.com/unoplatform/uno/issues/13384
• [ ] [iOS/Skia/macOS] RepeaterTests.VerifyRepeaterDoesNotLeakItemContainers (Skia started failing in https://github.com/unoplatform/uno/pull/16523)

In Progress

• [x] TestTextBoxReleaseOutFocused is flaky on WASM
• [x] TestTextBoxTap is very flaky on WASM
• [ ] Given_SymbolIcon.Validate_Size fails on all targets (Expected:<20>. Actual:<627.5>) (https://github.com/unoplatform/uno/pull/13069)
• [x] Android VerifyDropdownItemTemplateWithNoControl (failed cleanup System.Exception: Event was not raised.) (Youssef: passing for me - https://github.com/unoplatform/uno/pull/13077)
• [x] Android VerifyBreadcrumbDefaultAPIValues (failed cleanup System.Exception: Event was not raised.) (Youssef: passing for me - https://github.com/unoplatform/uno/pull/13077)
• [x] ValidateOverflowButtonHidesWhenAppropriateWithNoAppBarButtons, ValidateMoreButtonCanShowWithoutSizeChanging, ValidateOverflowButtonHidesWhenAppropriateWithPrimaryAppBarButtons are failing randomly on android. (Youssef: passing for me - https://github.com/unoplatform/uno/pull/13078)
• [x] Android/iOS When_EmbeddedResource (resource is not included when build on dotnet build) (Youssef: This appears to only be an issue for Uno.UI solution, ie, not a problem for external customers. Fixing in https://github.com/unoplatform/uno/pull/13079)
• [x] DoubleAnimation_Tests on android. These tests fail on android when running one by one, but don't when run with others. (Youssef: Can't repro. Enabling in #13082)

  System.Exception : Error while performing WaitFor([unknown], "Failed to find [Completed] on [Text]", null, null, null)
  ----> System.TimeoutException : Failed to find [Completed] on [Text]
  

• [x] When_Opacity_Paused_With_FillBehaviorHold_Then_Hold - on Android (Youssef: Can't repro. Enabling in #13082)
• [x] When_Transforms_Paused_With_FillBehaviorStop_Then_Hold. This test tends to fail when run alone, indicating a race condition, see in the comments for screenshots of the test. (Youssef: Can't repro. Enabling in #13082)

  1) Failed : SamplesApp.UITests.Windows_UI_Xaml_Media_Animation.DoubleAnimation_Tests.When_Transforms_Paused_With_FillBehaviorStop_Then_Hold
  ImageAssert.AreEqual @ line 85
  pixelTolerance: No color tolerance
  expected: Initial (When_Transforms_Paused_With_FillBehaviorStop_Then_Hold_Initial.png {Width=1280, Height=800}) in 
  {X=615,Y=442,Width=50,Height=50}
  actual  : Final (When_Transforms_Paused_With_FillBehaviorStop_Then_Hold_Final.png {Width=1280, Height=800}) in 
  {X=615,Y=442,Width=50,Height=50}
  

• [x] iOS/Android Given_ImageBrushStretch (Incorrect RenderTargetBitmap content) (youssef: Issue not related to RenderTargetBitmap. Fixing test in https://github.com/unoplatform/uno/pull/13081)

Done

• [x] https://github.com/unoplatform/uno/issues/10164
• [x] Android all Basics_AutomatedTransformation ( The specified child already has a parent. You must call removeView() on the child's parent first.) (https://github.com/unoplatform/uno/pull/13061)
• [x] Android/Skia When_Render_Then_CanRenderOnCanvas (Assert.Fail failed. ImageAssert.HasColorAt @ line 134) (works locally on Skia, fails on Android and being fixed in https://github.com/unoplatform/uno/pull/13070)
• [x] Wasm When_Reorder_To_Last_2 (https://github.com/unoplatform/uno/pull/13093)
• [x] [All platforms] RepeaterTests.VerifyUIElementsInItemsSource (https://github.com/unoplatform/uno/pull/13074)
• [x] [All platforms] RepeaterTests.NestedRepeaterWithDataTemplateScenario (https://github.com/unoplatform/uno/pull/13074)
• [x] [All platforms] RepeaterTests.VerifyStoreScenarioCache (https://github.com/unoplatform/uno/pull/13074)
• [x] https://github.com/unoplatform/uno/issues/10165 (https://github.com/unoplatform/uno/pull/13056)
• [x] https://github.com/unoplatform/uno/pull/13071

[Youssef1313]

When_Reorder_To_Last_2 on WASM, Android

This test appears to be Wasm only. I'm disabling in https://github.com/unoplatform/uno/pull/9571

[jeromelaban]

Images for When_Transforms_Paused_With_FillBehaviorStop_Then_Hold:

Teardown: Initial: Final:

[ramezgerges]

When_UsingMultipleLanguages is acting really off. First off, CultureInfo returns weird patterns e.g. in ft-CA, info.LongTimePattern returns HH 'h' mm instead of HH:mm. A possible hack is to replace the added garbage with a colon e.g. using Regex.Replace(value, " '[^']*' ", ":"), but obviously, that isn't ideal. Another thing is that our DateTimeFormatter implementation diverges a lot from the net7 implementation, so it might need a rewrite or something.

[ramezgerges]

VerifyFocusedItemIsRecycledOnCollectionReset seems to fail on skia due to a problem with dispatching measure/arrange. Items in an ItemsRepeater are only updated in a Measure, and it appears that no Measure is dispatched during the test (even though some paths are dirty for sure). A manual GetRoot(repeater).Measure(Window.Current.Bounds.Size) (where GetRoot just walks up the tree to get the RootVisual) passes the test, but I've failed to find the root of the problem.

Edit: It appears that TestServices.WindowHelper.WaitForIdle().Wait() is causing a deadlock (usually it's called with await instead of Wait()).

[Youssef1313]

I took a look at When_EmbeddedResource, I think the problem is related to shared projects.

The TargetPath here is SharedProjectEmbeddedFile.txt instead of Embedded/SubFolder1/SubFolder2/SharedProjectEmbeddedFile.txt.

[Youssef1313]

AssignLinkMetadata is somehow capable of determining the Link correctly. It should have run before AssignTargetPath.

We already have code for that, but it is racing with code in msbuild.

https://github.com/unoplatform/uno/blob/d1d468f240672e6ce8a2d06ad44db60c1b466fd3/src/SourceGenerators/Uno.UI.Tasks/Content/Uno.UI.Tasks.targets#L33-L40

https://github.com/dotnet/msbuild/blob/69a09e4d0ad2a26fab4100e78585c85bc2f9f5a7/src/Tasks/Microsoft.Common.CurrentVersion.targets#L3098-L3100

The evaluation from Uno.UI.Tasks.targets was done first (because it's imported first), then it got cleared by msbuild code from Microsoft.Common.CurrentVersion.targets