ci: MAUI integration tests

[vaind]

Integrates MAUI TestUtils, as mentioned in https://github.com/dotnet/maui/issues/3552#issuecomment-1129717167 and https://github.com/dotnet/xharness/issues/738

Some tests still fail, see e.g. https://github.com/getsentry/sentry-dotnet/pull/1703#issuecomment-1174010891 so I've enabled continue-on-error for the Android test run and I suggest creating a follow-up issue to resolve these failures.

closes #1704 closes #1662

#skip-changelog

[mattjohnsonpint]

Hi. I have another big PR with more MAUI stuff coming shortly that includes the deployment flag. But if you can find a way to get device tests going that would be amazing.

Currently, I can only write unit tests that run on net6.0 target, but there's a lot of code we'll want to tests that is only applicable under a specific device target, such as net6.0-android.

I think we want to do something like what the MAUI team is doing here: https://github.com/dotnet/maui/tree/main/src/TestUtils

Looks like others are trying this as well. See https://github.com/dotnet/maui/issues/3552#issuecomment-1129717167

[bruno-garcia]

Hi. I have another big PR with more MAUI stuff coming shortly that includes the deployment flag. But if you can find a way to get device tests going that would be amazing.

Currently, I can only write unit tests that run on net6.0 target, but there's a lot of code we'll want to tests that is only applicable under a specific device target, such as net6.0-android.

I think we want to do something like what the MAUI team is doing here: https://github.com/dotnet/maui/tree/main/src/TestUtils

Looks like others are trying this as well. See dotnet/maui#3552 (comment)

Captured this in this issue: https://github.com/getsentry/sentry-dotnet/issues/1704

[vaind]

I've tried integrating the MAUI TestUtils, as mentioned here https://github.com/dotnet/maui/issues/3552#issuecomment-1129717167 and here https://github.com/dotnet/xharness/issues/738

but am getting a lot of errors, such as Severity Code Description Project File Line Suppression State Error CS0234 The type or namespace name 'Graphics' does not exist in the namespace 'Microsoft.Maui' (are you missing an assembly reference?) TestUtils.DeviceTests (net6.0-android), TestUtils.DeviceTests (net6.0-ios), TestUtils.DeviceTests (net6.0-maccatalyst), TestUtils.DeviceTests (net6.0-windows10.0.19041.0) C:\dev\sentry-dotnet\MauiTestUtils\src\DeviceTests\AssertionExtensions.Android.cs 10 Active

I assume it has something to do with the imports I had to comment out in the imported projects, but I don't really know how to resolve that...

[mattjohnsonpint]

Thanks for getting this going!

I pushed some changes here that fix the build issues. I also converted tabs to spaces and moved the MauiTestUtils folder under the test folder to not clutter the root.

The tests seems to work when called via xharness:

xharness android test --output-directory=./test_output/net6.0-android/ --app=./test/MauiTestUtils/samples/DeviceTests.Sample/bin/Debug/net6.0-android/com.microsoft.maui.testutils.devicetests-Signed.apk --package-name=com.microsoft.maui.testutils.devicetests

I didn't try the other targets or wire it to CI, but this is a good start.

[vaind]

I've done a couple more fixes to make the Sentry.Maui.Device.TestApp work... However, after trying to integrate the test into the app, I'm getting the following errors:

Severity	Code	Description	Project	File	Line	Suppression State
Error	NETSDK1150	The referenced project '..\Sentry.Testing\Sentry.Testing.csproj' is a non self-contained executable.  A non self-contained executable cannot be referenced by a self-contained executable.  For more information, see https://aka.ms/netsdk1150	Sentry.Maui.Device.TestApp	C:\Program Files\dotnet\sdk\6.0.400-preview.22301.10\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.targets	1096	
Error	NETSDK1082	There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'android-x86'.	Sentry.Maui.Device.TestApp	C:\Program Files\dotnet\sdk\6.0.400-preview.22301.10\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets	430	
Error	NETSDK1082	There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'android-x64'.	Sentry.Maui.Device.TestApp	C:\Program Files\dotnet\sdk\6.0.400-preview.22301.10\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets	430	
Error	NETSDK1082	There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'android-arm64'.	Sentry.Maui.Device.TestApp	C:\Program Files\dotnet\sdk\6.0.400-preview.22301.10\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets	430	
Error	NETSDK1082	There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'android-arm'.	Sentry.Maui.Device.TestApp	C:\Program Files\dotnet\sdk\6.0.400-preview.22301.10\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets	430	

I understand there are AspNetCore-specific utilities in the Sentry.Testing project and that causes these errors. Do you think we should move these out to a separate project or is there a way to include them conditionally, when not building for net6.0-android?

aspnet maui support tracking issue: https://github.com/dotnet/aspnetcore/issues/35077

[mattjohnsonpint]

IMHO, we shouldn't have any ASP.NET Core dependencies in Sentry.Testing at all. That's supposed to be a common project for all test projects to use. I'll split them out.

[SimonCropp]

@vaind rebasing of develop will get you this change https://github.com/getsentry/sentry-dotnet/pull/1762

[vaind]

The current set of blockers:

• [x] MAUI platform detection support in Verify: https://github.com/VerifyTests/Verify/issues/550 - currently any test that uses Verify fails during static constructor of Namer
• [x] CI task test fails but doesn't give any error - no idea what's wrong - https://github.com/getsentry/sentry-dotnet/runs/7148454612?check_suite_focus=true
• [x] Windows build fails with

Error: C:\Users\runneradmin\.nuget\packages\microsoft.windowsappsdk\1.1.0\buildTransitive\Microsoft.Build.Msix.Packaging.targets(1479,5): error APPX1101: Payload contains two or more files with the same destination path 'Microsoft.TestPlatform.PlatformAbstractions.dll'. Source files:  [D:\a\sentry-dotnet\sentry-dotnet\test\Sentry.Maui.Device.TestApp\Sentry.Maui.Device.TestApp.csproj]
Error: C:\Users\runneradmin\.nuget\packages\microsoft.windowsappsdk\1.1.0\buildTransitive\Microsoft.Build.Msix.Packaging.targets(1479,5): error APPX1101: C:\Users\runneradmin\.nuget\packages\microsoft.testplatform.testhost\17.2.0\build\netcoreapp2.1\x64\Microsoft.TestPlatform.PlatformAbstractions.dll [D:\a\sentry-dotnet\sentry-dotnet\test\Sentry.Maui.Device.TestApp\Sentry.Maui.Device.TestApp.csproj]
Error: C:\Users\runneradmin\.nuget\packages\microsoft.windowsappsdk\1.1.0\buildTransitive\Microsoft.Build.Msix.Packaging.targets(1479,5): error APPX1101: C:\Users\runneradmin\.nuget\packages\microsoft.testplatform.objectmodel\17.2.0\lib\netcoreapp2.1\Microsoft.TestPlatform.PlatformAbstractions.dll [D:\a\sentry-dotnet\sentry-dotnet\test\Sentry.Maui.Device.TestApp\Sentry.Maui.Device.TestApp.csproj]
Error: C:\Users\runneradmin\.nuget\packages\microsoft.windowsappsdk\1.1.0\buildTransitive\Microsoft.Build.Msix.Packaging.targets(1479,5): error APPX1101: Payload contains two or more files with the same destination path 'xunit.runner.utility.netcoreapp10.dll'. Source files:  [D:\a\sentry-dotnet\sentry-dotnet\test\Sentry.Maui.Device.TestApp\Sentry.Maui.Device.TestApp.csproj]
Error: C:\Users\runneradmin\.nuget\packages\microsoft.windowsappsdk\1.1.0\buildTransitive\Microsoft.Build.Msix.Packaging.targets(1479,5): error APPX1101: C:\Users\runneradmin\.nuget\packages\xunit.runner.visualstudio\2.4.5\build\netcoreapp3.1\xunit.runner.utility.netcoreapp10.dll [D:\a\sentry-dotnet\sentry-dotnet\test\Sentry.Maui.Device.TestApp\Sentry.Maui.Device.TestApp.csproj]
Error: C:\Users\runneradmin\.nuget\packages\microsoft.windowsappsdk\1.1.0\buildTransitive\Microsoft.Build.Msix.Packaging.targets(1479,5): error APPX1101: C:\Users\runneradmin\.nuget\packages\xunit.runner.utility\2.4.1\lib\netcoreapp1.0\xunit.runner.utility.netcoreapp10.dll [D:\a\sentry-dotnet\sentry-dotnet\test\Sentry.Maui.Device.TestApp\Sentry.Maui.Device.TestApp.csproj]

[vaind]

🥳 android device tests: Tests run: 1023 Passed: 550 Inconclusive: 0 Failed: 472 Ignored: 1 https://github.com/getsentry/sentry-dotnet/runs/7163891114?check_suite_focus=true

[mattjohnsonpint]

Thanks for continuing the work on this! 🙂

IIRC, aren't there specific emulator images that are designed for headless operation? I thought I saw a few labeled that way when I was setting up my Android Studio on my dev machine.

[vaind]

ren't there specific emulator images that are designed for headless operation? I thought I saw a few labeled that way when I was setting up my Android Studio on my dev machine.

IIRC, aren't there specific emulator images that are designed for headless operation? I thought I saw a few labeled that way when I was setting up my Android Studio on my dev machine.

IDK, but why? The tests run just fine now in the emulator. The outstanding issue is with the standard test suite that fails for some reason on this branch, and of course some tests that fail on Android (we'll know a definitive number after Verify is updated)

[vaind]

Hi with the new Verify, it's Tests run: 1145 Passed: 1053 Inconclusive: 0 Failed: 91 Ignored: 1 which isn't so bad

[SimonCropp]

@vaind is it possible to see the failed tests in the build Action log?

[vaind]

@vaind is it possible to see the failed tests in the build Action log?

I've checked the xharness XML output and it said "xunit" in there so I assumed it's that format but it doesn't seem to be the case (opening with npm xunit-viewer doesn't work neither does the test-reported GHA.

I could simply grep the failed test cases, but until I do that, maybe there's a parser that could give more info - does anyone know what this format might be? Sample:

<?xml version="1.0" encoding="utf-8"?>
<assemblies>
  <assembly name="Sentry.Maui.Device.TestApp.dll" environment="64-bit .NET  [collection-per-class, non-parallel]" test-framework="xUnit.net 2.4.1.0" run-date="2022-07-02" run-time="19:12:55" total="8" passed="5" failed="2" skipped="1" time="5.895" errors="0">
    <errors />
    <collection total="8" passed="5" failed="2" skipped="1" name="Test collection for Sentry.Maui.Device.TestApp.UnitTests" time="0.544">
      <test name="OutputTest" type="Sentry.Maui.Device.TestApp.UnitTests" method="OutputTest" time="0.390377" result="Pass">
        <output><![CDATA[This is test output.
]]></output>
      </test>
      <test name="SkippedTest" type="Sentry.Maui.Device.TestApp.UnitTests" method="SkippedTest" time="0" result="Skip">
        <reason><![CDATA[This test is skipped.]]></reason>
      </test>
      <test name="SuccessfulTest" type="Sentry.Maui.Device.TestApp.UnitTests" method="SuccessfulTest" time="0.009448" result="Pass" />
      <test name="FailingTest" type="Sentry.Maui.Device.TestApp.UnitTests" method="FailingTest" time="0.110607" result="Fail">
        <failure exception-type="System.Exception">
          <message><![CDATA[System.Exception : This is meant to fail.]]></message>
          <stack-trace><![CDATA[   at Sentry.Maui.Device.TestApp.UnitTests.FailingTest()
   at System.Reflection.RuntimeMethodInfo.Invoke(Object , BindingFlags , Binder , Object[] , CultureInfo )]]></stack-trace>
        </failure>
      </test>
      <test name="FailingOutputTest" type="Sentry.Maui.Device.TestApp.UnitTests" method="FailingOutputTest" time="0.006816" result="Fail">
        <output><![CDATA[This is test output.
]]></output>
        <failure exception-type="System.Exception">
          <message><![CDATA[System.Exception : This is meant to fail.]]></message>
          <stack-trace><![CDATA[   at Sentry.Maui.Device.TestApp.UnitTests.FailingOutputTest()
   at System.Reflection.RuntimeMethodInfo.Invoke(Object , BindingFlags , Binder , Object[] , CultureInfo )]]></stack-trace>
        </failure>
      </test>
      <test name="ParameterizedTest(number: 1)" type="Sentry.Maui.Device.TestApp.UnitTests" method="ParameterizedTest" time="0.026247" result="Pass" />
      <test name="ParameterizedTest(number: 2)" type="Sentry.Maui.Device.TestApp.UnitTests" method="ParameterizedTest" time="0.000163" result="Pass" />
      <test name="ParameterizedTest(number: 3)" type="Sentry.Maui.Device.TestApp.UnitTests" method="ParameterizedTest" time="0.000141" result="Pass" />
    </collection>
  </assembly>
</assemblies>

[SimonCropp]

looks like this https://xunit.net/docs/format-xml-v2 ?

[SimonCropp]

wading though the xharness code. it seems it is configurable https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.TestRunners.Xunit/XUnitTestRunner.cs#L927

See https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.Common/XmlResultJargon.cs

and the arg is https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.CLI/CommandArguments/Apple/Arguments/XmlResultJargonArgument.cs

[vaind]

looks like this https://xunit.net/docs/format-xml-v2 ?

indeed, it seems like xunit v2, that's why it didn't work with the xunit parser I've tried. A quick search didn't yield any other so I'll just make a simple script to parse it and print to GitHub Actions results

[vaind]

wading though the xharness code. it seems it is configurable https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.TestRunners.Xunit/XUnitTestRunner.cs#L927

See https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.Common/XmlResultJargon.cs

and the arg is https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.CLI/CommandArguments/Apple/Arguments/XmlResultJargonArgument.cs

that only seems to work for iOS, but it helped me find the Android alternative, which is setting a NUNIT_XML_VERSION environment variable - however, doing that changes nothing about the output.

[vaind]

https://github.com/getsentry/sentry-dotnet/actions/runs/2611409713#summary-7184365508

Summary

Assembly	Passed	Failed	Skipped
Sentry.Tests.dll	1079	36	0
Sentry.Maui.Tests.dll	21	1	0

and some are still coming from Verify internals, such as

• SerializationTests.Serialization(name: "string", target: "string value") - Failed

System.IO.IOException : Read-only file system : '/home/runner/work/sentry-dotnet/sentry-dotnet/test/Sentry.Tests'
at System.IO.FileSystem.CreateDirectory(String )
   at System.IO.Directory.CreateDirectory(String )
   at InnerVerifier..ctor(String sourceFile, VerifySettings settings, GetFileConvention fileConvention)
   at VerifyXunit.Verifier.GetVerifier(VerifySettings settings, String sourceFile)
   at VerifyXunit.Verifier.<>c__DisplayClass1_0.<<Verify>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at SerializationTests.Serialization(String name, Object target)
--- End of stack trace from previous location ---

[SimonCropp]

why does this add two CategoryDiscoverer classes

[vaind]

Anyone reviewing this, please ignore *.cs code under MauiTestUtils - that's code copied from MAUI and we don't want to touch that unless absolutely necessary.

[mattjohnsonpint]

Yay, it works! 🎉