Steeltoe
Steeltoe copied to clipboard
Update to xUnit v3 on Microsoft Testing Platform v2
Description
This PR migrates existing xUnit v3 tests to run on Microsoft Testing Platform v2, instead of VSTest. This enables running tests both via dotnet test and as direct executables.
What's changed
- Replaced coverlet with
Microsoft.Testing.Extensions.CodeCoverage, because it is incompatible- Removed Coverlet workaround
- Removed
xunit.runner.visualstudioandMicrosoft.NET.Test.Sdk, which are no longer needed - Alternate libraries used for hang/crashdump and TRX reports
- Added
global.json, which is required for MTP v2 when using the .NET 10 SDK - Different file format for coverage settings (
.runsettingsreplaced bycoverage.config) - Different coverage output format (OpenCover is no longer available, Cobertura can't be read by Sonar, so what's left is Visual Studio XML format)
- Different command-line arguments to filter tests and take crash/hangdumps
- In CI build, run tests against all target frameworks in a single command (because
TestTfmsInParallelnow actually works) - Make xUnit use MTP2 via
UseMicrosoftTestingPlatformRunner - Keep using
xunit.runner.jsoninstead oftestconfig.json, because it works with multiple runners - Not using the superior GitHubActionsTestLogger, due to license terms about Russia/Ukraine
- When no tests are found,
dotnet testexists with an error (can be overruled with--minimum-expected-tests 0or catching the related exit code) - We lost branch coverage and risk hotspots, because ReportGenerator does not support it with VS coverage format
- The CI build log does not tell which test(s) failed (only the assembly), and there is no stacktrace. Need to search for it on the GitHub Actions summary tab
Verification
- Tests are discovered (5520 in total, 1840 per target framework)
- [x] In Visual Studio 2026 Test Explorer
- [x] In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
- [x] In Visual Studio 2026 R# Unit Tests
- [x] In
dotnet teston the command line
- Tests can be run
- [x] In Visual Studio 2026 Test Explorer
- [x] In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
- [x] In Visual Studio 2026 R# Unit Tests
- [x] In
dotnet teston the command line - [x] Test outcomes are listed on the GitHub Actions summary tab
- Tests can be debugged
- [x] In Visual Studio 2026 Test Explorer
- [x] In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04 :raised_hand: It's only possible to debug once; a VS restart is required to debug again (tracked at https://github.com/microsoft/testfx/issues/7043)
- [x] In Visual Studio 2026 R# Unit Tests
- All tests succeed
- [x] In Visual Studio 2026 Test Explorer
- [x] In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
- [x] In Visual Studio 2026 R# Unit Tests
- [x] In
dotnet teston the command line - [x] In CI build on Ubuntu/Windows/macOS, split into regular/memorydump testruns
- Coverage is reported
- [ ] Coverage files contain source-link paths
- [x] Coverage on PR diff in SonarCloud and Sonar PR comment
- [x] Coverage on full branch displayed on GitHub Actions summary tab and PR comment (only Steeltoe libraries)
:raised_hand: We lost statement branch coverage here
- [x] Coverage on full branch roughly matches outcome from main branch (only Steeltoe libraries)
:raised_hand: We lost statement branch coverage here
- Test failures
- [x] Crashdump is captured and uploaded during CI build :raised_hand: Works, but outputs a 160 MB text log, which makes the GitHub Actions UI completely unresponsive; tracked at https://github.com/microsoft/testfx/issues/7045
- [ ] Hangdump is captured and uploaded during CI build :raised_hand: This doesn't currently work, tracked at https://github.com/microsoft/testfx/issues/6776
- [x] Failed test is reported
- [x] In Visual Studio 2026 Test Explorer
- [x] In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
- [x] In Visual Studio 2026 R# Unit Tests
- [x] In
dotnet teston the command line - [x] On GitHub Actions summary tab
- [x] In CI build log
- [x] Skipped test is reported
- [x] In Visual Studio 2026 Test Explorer
- [x] In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
- [x] In Visual Studio 2026 R# Unit Tests
- [x] In
dotnet teston the command line - [x] On GitHub Actions summary tab
- [x] In CI build log
Quality checklist
- [ ] Your code complies with our Coding Style.
- [ ] You've updated unit and/or integration tests for your change, where applicable.
- [ ] You've updated documentation for your change, where applicable. If your change affects other repositories, such as Documentation, Samples and/or MainSite, add linked PRs here.
- [ ] There's an open issue for the PR that you are making. If you'd like to propose a new feature or change, please open an issue to discuss the change or find an existing issue.
- [ ] You've added required license files and/or file headers (explaining where the code came from with proper attribution), where code is copied from StackOverflow, a blog, or OSS.
Summary - All Code Coverage (ubuntu-latest)
| Assembly | Line coverage | Branch coverage |
|---|---|---|
| Steeltoe.Bootstrap.AutoConfiguration | 97.4% | 100% |
| Steeltoe.Common | 84.5% | 77.7% |
| Steeltoe.Common.Certificates | 96.2% | 85.7% |
| Steeltoe.Common.Hosting | 84% | 65% |
| Steeltoe.Common.Http | 100% | 82.3% |
| Steeltoe.Common.Logging | 81.3% | 41.6% |
| Steeltoe.Common.Net | 64.5% | 66.6% |
| Steeltoe.Configuration.Abstractions | 98.5% | 91.6% |
| Steeltoe.Configuration.CloudFoundry | 99.1% | 92.3% |
| Steeltoe.Configuration.ConfigServer | 97.4% | 92.2% |
| Steeltoe.Configuration.Encryption | 97.7% | 92.4% |
| Steeltoe.Configuration.Kubernetes.ServiceBindings | 93.3% | 87.5% |
| Steeltoe.Configuration.Placeholder | 91.9% | 78.8% |
| Steeltoe.Configuration.RandomValue | 93.2% | 87.5% |
| Steeltoe.Configuration.SpringBoot | 98.3% | 95% |
| Steeltoe.Connectors | 94.3% | 88.7% |
| Steeltoe.Connectors.EntityFrameworkCore | 81.5% | 75% |
| Steeltoe.Discovery.Configuration | 92.8% | 100% |
| Steeltoe.Discovery.Consul | 97.8% | 96.2% |
| Steeltoe.Discovery.Eureka | 91.5% | 84.5% |
| Steeltoe.Discovery.HttpClients | 95% | 95.8% |
| Steeltoe.Logging.Abstractions | 99.4% | 97% |
| Steeltoe.Logging.DynamicConsole | 100% | 96.8% |
| Steeltoe.Logging.DynamicSerilog | 99.1% | 95.8% |
| Steeltoe.Management.Abstractions | 100% | 100% |
| Steeltoe.Management.Endpoint | 95.5% | 89.3% |
| Steeltoe.Management.Prometheus | 95.9% | 92.3% |
| Steeltoe.Management.Tasks | 100% | **** |
| Steeltoe.Management.Tracing | 100% | 83.3% |
| Steeltoe.Security.Authentication.JwtBearer | 100% | 100% |
| Steeltoe.Security.Authentication.OpenIdConnect | 74.4% | 57.6% |
| Steeltoe.Security.Authorization.Certificate | 96.3% | 80% |
| Steeltoe.Security.DataProtection.Redis | 100% | **** |
Quality Gate passed
Issues
0 New issues
0 Accepted issues
Measures
1 Security Hotspot
0.0% Coverage on New Code
0.0% Duplication on New Code