uno
uno copied to clipboard
ci: Add macOS desktop runtime tests
GitHub Issue (If applicable): closes #
PR Type
What kind of change does this PR introduce?
What is the current behavior?
What is the new behavior?
PR Checklist
Please check if your PR fulfills the following requirements:
- [ ] Docs have been added/updated which fit documentation template (for bug fixes / features)
- [ ] Unit Tests and/or UI Tests for the changes have been added (for bug fixes / features) (if applicable)
- [ ] Validated PR
Screenshots Compare Test Runresults. - [ ] Contains NO breaking changes
- [ ] Associated with an issue (GitHub or internal) and uses the automatic close keywords.
- [ ] Commits must be following the Conventional Commits specification.
Other information
Internal Issue (If applicable):
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
❌ (F) Check_FontFallback_Shaping()
EXCEPTION>Images have different resolutions.
expected:(98,34)
actual :(97,34)
The actual image is not the same as the expected one.
actual RSMD: 0.1720096541053666
threshold: 0.15
Looks like a minor difference, where the bitmap's width is smaller, by a pixel, than expected. That test is already ignored for X11.
screenshot1.png
screenshot2.png
I think we should bump the threshold to 0.18 to cover the difference. Also we can remove the separate width/height checks since the ImageAssert.AreSimilarAsync already does it.
The alternative would be to ignore the test for macOS (like X11) but without testing it we run into the risk of larger regressions.
@Youssef1313 ?
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
❌ (F) When_Large_List_Scroll_To_End_Then_Back_Up_TryClick()
EXCEPTION>Assert.Fail failed. ImageAssert.HasColorAt @ line 1991
====================
Color at (70,65) is not expected
expected: FF1A69A6 #FF1A69A6
actual : FF1A6AA7 #FF1A6AA7
tolerance : 0
difference: 1
screenshot size: 100x500
Also looks like a minor difference and pass if the tolerance is set to 1.
@Youssef1313 ? @ramezgerges ?
Screenshot
The other failures all seems to be keyboard related, mostly because the shortcuts between macOS and Windows are different by design.
I'll be fixing them in upcoming commits...
❌ (F) When_Large_List_Scroll_To_End_Then_Back_Up_TryClick() EXCEPTION>Assert.Fail failed. ImageAssert.HasColorAt @ line 1991 ==================== Color at (70,65) is not expected expected: FF1A69A6 #FF1A69A6 actual : FF1A6AA7 #FF1A6AA7 tolerance : 0 difference: 1 screenshot size: 100x500Also looks like a minor difference and pass if the
toleranceis set to1.@Youssef1313 ? @ramezgerges ?
This is a bit weird, why would it be different between platforms (unless there's a bug in the metal backend in skia)? I would try testing this with skiasharp 3 and see if that fixes things. SK3 already fixes a bunch of linux-specific bugs. Anyway, I guess adding a tolence of 1 when MACOS is fine.
For Check_FontFallback_Shaping, the one pixel difference is a bug that is acceptable currently. Space widths can differ from font to another, and we may use wrong font to determine space width in some cases.
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
🤖 Your Docs stage site is ready! Visit it here: https://unodocsprstaging.z13.web.core.windows.net/pr-17333/index.html
🤖 Your WebAssembly Sample App stage site is ready! Visit it here: https://unowasmprstaging.z20.web.core.windows.net/pr-17333/index.html
The build 146184 found UI Test snapshots differences: android-28-net8: 18, android-28-net8-Snap: 43, ios: 8, ios-Snap: 57, skia-gtk-screenshots: 67, skia-linux-screenshots: 68, skia-windows-screenshots: 64, wasm: 455, wasm-automated-net9.0-UWP-Default-automated: 26, wasm-automated-net9.0-UWP-RuntimeTests-0: 0, wasm-automated-net9.0-UWP-RuntimeTests-1: 0, wasm-automated-net9.0-UWP-RuntimeTests-2: 0, wasm-automated-net9.0-WinUI-Benchmarks-automated: 0, wasm-automated-net9.0-WinUI-RuntimeTests-0: 0, wasm-automated-net9.0-WinUI-RuntimeTests-1: 0, wasm-automated-net9.0-WinUI-RuntimeTests-2: 0
Details
-
android-28-net8: 18 changed over 829🚨🚨 Comparison Details (first 20) 🚨🚨
ComboBoxTests_Fullscreen_Popup_Generic_OpenedComboBoxTests_Fullscreen_Popup_Generic_SamplesApp_Wasm_Windows_UI_Xaml_Controls_ComboBox_ComboBox_FullScreen_PopupDecimalFormatterTest_UITests_Shared_Microsoft_UI_Xaml_Controls_NumberBoxTests_NumberBoxPageDetereminate_ProgressRing_Validation50_[#FF0000_#008000_#008000_#FF0000]_Progress-Ring-Value-50Detereminate_ProgressRing_Validation25_[#FF0000_#008000_#FF0000_#FF0000]_Progress-Ring-Value-25When_NoSelectionWithItemClick_Then_PointersEvents_UITests_Windows_UI_Xaml_Controls_ListView_ListView_Selection_PointersWhen_NoSelection_Then_PointersEvents_UITests_Windows_UI_Xaml_Controls_ListView_ListView_Selection_PointersWhen_SingleSelectionWithItemClick_Then_PointersEvents_UITests_Windows_UI_Xaml_Controls_ListView_ListView_Selection_PointersDetereminate_ProgressRing_Validation75_[#FF0000_#008000_#008000_#008000]_Progress-Ring-Value-75WebView_NavigateToAnchor_InitialFocusManager_GetFocusedElement_Rectangle_Validation_Uno_UI_Samples_Content_UITests_FocusTests_FocusManager_GetFocus_AutomatedListView_SelectedItems_SamplesApp_Windows_UI_Xaml_Controls_ListView_ListViewSelectedItemsProgressRing_Visibility_Collapsed_UITests_Windows_UI_Xaml_Controls_ProgressRing_WindowsProgressRing_GH1220SequentialAnimations_SamplesApp_Windows_UI_Xaml_Media_Animation_SequentialAnimationsPageWhen_Parent_PointerMoved_After_drag_on_non-scrolling_ScrollViewerWhen_Parent_PointerMoved_After_drag_on_ScrollViewer_-_touchProgressRing_IsEnabled_Running_UITests_Windows_UI_Xaml_Controls_ProgressRing_WindowsProgressRing_GH1220UpDownEnabledTest_UITests_Shared_Microsoft_UI_Xaml_Controls_NumberBoxTests_NumberBoxPage
-
android-28-net8-Snap: 43 changed over 1067🚨🚨 Comparison Details (first 20) 🚨🚨
MUX_NumberBox_UITests_Shared_Microsoft_UI_Xaml_Controls_NumberBoxTests_NumberBoxPageNavigationView_MUXControlsTestApp_NavigationViewTopNavOnlyPage_MUXControlsTestApp_NavigationViewTopNavOnlyPageNavigationView_MUXControlsTestApp_NavigationViewTopNavPage_MUXControlsTestApp_NavigationViewTopNavPageTextBox_Input_InputScope_PhoneNumber_Uno_UI_Samples_Content_UITests_TextBoxControl_Input_InputScope_PhoneNumberTextBox_Input_InputScope_Search_Uno_UI_Samples_Content_UITests_TextBoxControl_Input_InputScope_SearchTextBox_Input_InputScope_Url_Uno_UI_Samples_Content_UITests_TextBoxControl_Input_InputScope_UrlTextBox_Input_Multiline_AutoHeight_Uno_UI_Samples_Content_UITests_TextBoxControl_Input_Multiline_AutoHeightBorder_Border_CornerRadius_Gradient_UITests_Windows_UI_Xaml_Controls_BorderTests_Border_CornerRadius_GradientListView_ListViewSelectedItems_SamplesApp_Windows_UI_Xaml_Controls_ListView_ListViewSelectedItemsListView_ListView_With_ListViews_Count_Measure_UITests_Shared_Windows_UI_Xaml_Controls_ListView_ListView_With_ListViews_Count_MeasureButtons_AppBar_KeyBoard_Uno_UI_Samples_Content_UITests_ButtonTestsControl_AppBar_KeyBoardTabView_UITests_Microsoft_UI_Xaml_Controls_TabViewTests_TabViewPage_UITests_Microsoft_UI_Xaml_Controls_TabViewTests_TabViewPageTeachingTip_UITests_Microsoft_UI_Xaml_Controls_TeachingTipTests_TeachingTipPage_UITests_Microsoft_UI_Xaml_Controls_TeachingTipTests_TeachingTipPageBorder_AutoBorderStretchwithtopmargin_Uno_UI_Samples_UITests_BorderTestsControl_AutoBorderStretchwithtopmarginBorder_Border_CornerRadius_Uno_UI_Samples_UITests_BorderTestsControl_Border_CornerRadiusButtons_AppBarButtonTest_Uno_UI_Samples_Content_UITests_ButtonTestsControl_AppBarButtonTestGesture_Recognizer_Pointer_Events_test_bench_UITests_Shared_Windows_UI_Input_GestureRecognizer_PointersEventsNavigationView_MUXControlsTestApp_NavigationViewCustomThemeResourcesPage_MUXControlsTestApp_NavigationViewCustomThemeResourcesPageTextBlock_TextBlock_LineHeight_Multiline_Uno_UI_Samples_Content_UITests_TextBlockControl_TextBlock_LineHeight_MultilineBorder_Border_CornerRadius_Binding_Uno_UI_Samples_UITests_BorderTestsControl_Border_CornerRadius_Binding
-
ios: 8 changed over 267🚨🚨 Comparison Details (first 20) 🚨🚨
TextBox_UpdatedBinding_On_OneWay_Mode_UITests_Windows_UI_Xaml_Controls_TextBox_TextBox_BindingsWhen_NoSelection_Then_PointersEvents_UITests_Windows_UI_Xaml_Controls_ListView_ListView_Selection_PointersWhen_SingleSelectionWithItemClick_Then_PointersEvents_UITests_Windows_UI_Xaml_Controls_ListView_ListView_Selection_PointersCheck_ListView_Swallows_Measure_UITests_Shared_Windows_UI_Xaml_Controls_ListView_ListView_With_ListViews_Count_MeasureWhen_MultipleSelectionWithoutItemClick_Then_PointersEvents_UITests_Windows_UI_Xaml_Controls_ListView_ListView_Selection_PointersWhen_NoSelectionWithItemClick_Then_PointersEvents_UITests_Windows_UI_Xaml_Controls_ListView_ListView_Selection_PointersProgressRing_IsEnabled_Running_UITests_Windows_UI_Xaml_Controls_ProgressRing_WindowsProgressRing_GH1220ProgressRing_Visibility_Collapsed_UITests_Windows_UI_Xaml_Controls_ProgressRing_WindowsProgressRing_GH1220
-
ios-Snap: 57 changed over 1058🚨🚨 Comparison Details (first 20) 🚨🚨
Brushes_ImageBrushWithScaleTransform_Uno_UI_Samples_UITests_ImageBrushTestControl_ImageBrushWithScaleTransformContentControl_ContentControl_UnsetContent_GenericApp_Views_Content_UITests_ContentControlTestsControl_ContentControl_UnsetContentNavigationView_MUXControlsTestApp_NavigationViewMinimalPage_MUXControlsTestApp_NavigationViewMinimalPageBrushes_RectangleStretchFill_Uno_UI_Samples_UITests_ImageBrushTestControl_RectangleStretchFillIcons_UITests_Shared_Windows_UI_Xaml_Controls_BitmapIconTests_BitmapIcon_Foreground_UITests_Shared_Windows_UI_Xaml_Controls_BitmapIconTests_BitmapIcon_ForegroundListView_ListViewGroupedChanging_SamplesApp_Windows_UI_Xaml_Controls_ListView_ListViewGroupedChangingNavigationView_MUXControlsTestApp_PaneLayoutTestPage_MUXControlsTestApp_PaneLayoutTestPageSwipeControl_MUXControlsTestApp_SwipeControlPage2_MUXControlsTestApp_SwipeControlPage2TextBlock_TextBoxSizeChanging_Uno_UI_Samples_Content_UITests_TextBlockControl_TextBoxSizeChangingBrushes_ImageBrushWithCompositeTransform_Uno_UI_Samples_UITests_ImageBrushTestControl_ImageBrushWithCompositeTransformGrid_Grid_DataBound_ColumnRow_Definitions_Uno_UI_Samples_Content_UITests_GridTestsControl_Grid_DataBound_ColumnRow_DefinitionsNavigationView_MUXControlsTestApp_NavigationViewBlankPage1_MUXControlsTestApp_NavigationViewBlankPage1NavigationView_MUXControlsTestApp_NavigationViewCompactPaneLengthTestPage_MUXControlsTestApp_NavigationViewCompactPaneLengthTestPageNavigationView_MUXControlsTestApp_NavigationViewCustomThemeResourcesPage_MUXControlsTestApp_NavigationViewCustomThemeResourcesPageNavigationView_MUXControlsTestApp_NavigationViewRS4Page_MUXControlsTestApp_NavigationViewRS4PageBrushes_ImageBrushStretch_Uno_UI_Samples_UITests_ImageBrushTestControl_ImageBrushStretchClipping_SamplesApp_Windows_UI_Xaml_Clipping_XamlButtonWithClipping_Scrollable_SamplesApp_Windows_UI_Xaml_Clipping_XamlButtonWithClipping_ScrollableListView_ListViewHeaderFooter_ImplicitlyBinded_SamplesApp_Windows_UI_Xaml_Controls_ListView_ListViewHeaderFooter_ImplicitlyBindedBrushes_ImageBrush_Stretch_None_Uno_UI_Samples_UITests_ImageBrushTestControl_ImageBrush_Stretch_NoneFocus_UITests_Windows_UI_Xaml_FocusTests_Focus_FocusCycle_UITests_Windows_UI_Xaml_FocusTests_Focus_FocusCycle
-
skia-gtk-screenshots: 67 changed over 2224🚨🚨 Comparison Details (first 20) 🚨🚨
Focus_FocusVisual_Properties.png-darkBorderImageBrushRelativeTransform.png-darkButtons.png-darkImageBrush_SameWithDelay.pngButtonClippingTestsControl.png-darkButtonClippingTestsControl.pngClipboardTests.png-darkBorderImageBrushRelativeTransform.pngButtons.pngDisplayInformation.png-darkFocus_FocusVisual_Properties.pngClipboardTests.pngImageBrushAlignmentXY2.png-darkTransformToVisual_Simple.png-darkTransformToVisual_Simple.pngPopup_Simple.png-darkPopup_Simple.pngDisplayInformation.pngImageBrush_SameWithDelay.png-darkPopup_HVAlignments.png-dark
-
skia-linux-screenshots: 68 changed over 2224🚨🚨 Comparison Details (first 20) 🚨🚨
BorderImageBrush.png-darkCalendarView_Theming.png-darkDropDownButtonPage.png-darkButtons.png-darkButtons.pngCompositionEffectBrush.png-darkCompositionEffectBrush.pngExamples.pngBorderImageBrushRelativeTransform.png-darkClipboardTests.pngDoubleImageBrushInList.png-darkDoubleImageBrushInList.pngImageBrushInList.png-darkImageBrushInList.pngGamepad_CurrentReading.png-darkGamepad_CurrentReading.pngImage_UseTargetSize.png-darkImage_UseTargetSize.pngBorderImageBrushRelativeTransform.pngGamepad_Enumeration.png-dark
-
skia-windows-screenshots: 64 changed over 2224🚨🚨 Comparison Details (first 20) 🚨🚨
AppWindowPositionAndSize.png-darkFocus_FocusVisual_Properties.png-darkAppWindowPositionAndSize.pngDisplayInformation.png-darkDisplayInformation.pngDropDownButtonPage.png-darkDropDownButtonPage.pngGamepad_Enumeration.png-darkGamepad_Enumeration.pngButtonClippingTestsControl.pngButtons.png-darkCalendarView_Theming.png-darkCalendarView_Theming.pngButtons.pngClipboardTests.png-darkClipboardTests.pngColorPickerSample.png-darkColorPickerSample.pngExamples.png-darkExamples.png
-
wasm: 455 changed over 1044🚨🚨 Comparison Details (first 20) 🚨🚨
MUXControlsTestApp.NavigationViewMenuItemStretchPageMUXControlsTestApp.NavigationViewStretchPageMUXControlsTestApp.NavigationViewTopNavPageSamplesApp.Samples.RoutedEvents.RoutedEventsPageSamplesApp.Wasm.Windows_UI_Xaml_Controls.ComboBox.ComboBox_DisplayMemberPathSamplesApp.Windows_UI_Xaml.Clipping.DoubleAnimationClippingSamplesApp.Windows_UI_Xaml.Clipping.PanelChildrenClippingSamplesApp.Windows_UI_Xaml_Controls.ListView.ListViewEmptyGroupsSamplesApp.Windows_UI_Xaml_Shapes.PathTestsControl.PathCircularInsideEllipseUITests.Microsoft_UI_Xaml_Controls.TabViewTests.TabViewBasicPageUITests.Shared.Windows_ApplicationModel.Calls.PhoneCallManagerTestsUITests.Shared.Windows_Devices.Midi.MidiDeviceEnumerationTestsUITests.Shared.Windows_UI_Input.GestureRecognizerTests.RightTappedTestsUITests.Shared.Windows_UI_Xaml_Controls.BitmapIconTests.BitmapIcon_ForegroundUITests.Shared.Windows_UI_Xaml_Controls.ListView.ListView_Aligned_LeftUITests.Shared.Windows_UI_Xaml_Media.Transform.BasicsUITests.Windows_UI_Xaml_Controls.CommandBar.BackButtonTitle.CommandBar_FrameUITests.Windows_UI_Xaml_Controls.CommandBar.CommandBar_Native_FrameUITests.Windows_UI_Xaml_Controls.CommandBar.CommandBar_Native_With_AppBarButton_With_ForegroundUITests.Windows_UI_Xaml_Controls.RelativePanelTests.RelativePanel_AlignHorizontalVerticalCenter
-
wasm-automated-net9.0-UWP-Default-automated: 26 changed over 1947🚨🚨 Comparison Details (first 20) 🚨🚨
Default_Uno_UI_Samples_Content_UITests_WebView_WebView_AnchorNavigation_Uno_UI_Samples_Content_UITests_WebView_WebView_AnchorNavigationImageStretch_None_Uno_UI_Samples_UITests_ImageTestsControl_Image_Stretch_NoneAutomation_UITests_Shared_Windows_UI_Xaml_Automation_AutomationProperties_Name_UITests_Shared_Windows_UI_Xaml_Automation_AutomationProperties_NameListView_ListViewSelectedItems_SamplesApp_Windows_UI_Xaml_Controls_ListView_ListViewSelectedItemsDefault_StrokeThickness_MyRectListView_SelectedItems_SamplesApp_Windows_UI_Xaml_Controls_ListView_ListViewSelectedItemsBrushes_Uno_UI_Samples_UITests_ImageBrushTestControl_Ellipse_Uno_UI_Samples_UITests_ImageBrushTestControl_EllipseExpander_WinUIExpanderPage_UITests_Shared_Microsoft_UI_Xaml_Controls_ExpanderTests_WinUIExpanderPageDefault_StrokeThickness_MyPathBrushes_BorderImageBrush_Uno_UI_Samples_UITests_ImageBrushTestControl_BorderImageBrushDefault_SamplesApp_Wasm_Windows_UI_Xaml_Controls_ListView_ListView_IsSelected_SamplesApp_Wasm_Windows_UI_Xaml_Controls_ListView_ListView_IsSelectedPointers_UITests_Windows_UI_Input_PointersTests_HitTest_GeometryGroup_UITests_Windows_UI_Input_PointersTests_HitTest_GeometryGroupMediaPlayerElement_Mini_player_UITests_Shared_Windows_UI_Xaml_Controls_MediaPlayerElement_MediaPlayerElement_MinimalMediaPlayerElement_Sources_UITests_Shared_Windows_UI_Xaml_Controls_MediaPlayerElement_MediaPlayerElement_SourcesMediaPlayerElement_Using_3gp_UITests_Shared_Windows_UI_Xaml_Controls_MediaPlayerElement_MediaPlayerElement_3gp_ExtensionMediaPlayerElement_Using_mp3_Audio_only_UITests_Shared_Windows_UI_Xaml_Controls_MediaPlayerElement_MediaPlayerElement_Mp3_ExtensionMediaPlayerElement_Using_ogg_UITests_Shared_Windows_UI_Xaml_Controls_MediaPlayerElement_MediaPlayerElement_Ogg_ExtensionListView_ListView_With_ListViews_Count_Measure_UITests_Shared_Windows_UI_Xaml_Controls_ListView_ListView_With_ListViews_Count_MeasureSequentialAnimations_SamplesApp_Windows_UI_Xaml_Media_Animation_SequentialAnimationsPageWhen_NoSelectionWithItemClick_Then_PointersEvents_UITests_Windows_UI_Xaml_Controls_ListView_ListView_Selection_Pointers
-
wasm-automated-net9.0-UWP-RuntimeTests-0: 0 changed over 1 -
wasm-automated-net9.0-UWP-RuntimeTests-1: 0 changed over 1 -
wasm-automated-net9.0-UWP-RuntimeTests-2: 0 changed over 1 -
wasm-automated-net9.0-WinUI-Benchmarks-automated: 0 changed over 1 -
wasm-automated-net9.0-WinUI-RuntimeTests-0: 0 changed over 1 -
wasm-automated-net9.0-WinUI-RuntimeTests-1: 0 changed over 1 -
wasm-automated-net9.0-WinUI-RuntimeTests-2: 0 changed over 1