dotnet-test-nunit
dotnet-test-nunit copied to clipboard
Playlists wrongly group tests
dotnet-test-nunit v3.4.0-beta-3, with NUnit 3 Test Adapter (on Visual Studio 2015 Update 3 Community Edition)
The project under test is ASP.Net Core targeting .Net 4.5.2 (i.e. not a "pure" dotnet project).
Creating a playlist with 20 tests (out of 100) by selecting in Test Explorer, right-click, Add To Playlist > New Playlist. Only these 20 tests show in Test Explorer. Now run "All Tests".
Result:
The first 13 tests show in green, the next 7 tests are dimmed out (normally this indicates they were not run).
At the bottom of the screen it says "18 tests passed".
See screenshot
What should have happened: All 20 tests are highlighted green. At the bottom it says "20 tests passed".
Workaround: Do not use playlists. The default view with all tests, grouped by "outcome" works fine. See also other issue I submitted - it seems that the only grouping which works properly is the default "by outcome" view with "all tests".
Update Edit:
-
I found that in addition to above the playlist file contains OTHER tests than what I had selected in the UI.
-
And, reversely, the UI shows other tests than what the playlist contains. Example - Making a short playlist with two tests. File then, for example contains:
<Playlist Version="1.0"><Add Test="StatsServicesTest.AreaTests.AreaRetrieve_MustRetrieve" /><Add Test="StatsServicesTest.AreaTests.AreaRetrieve_ProperData" /></Playlist>
When filtering the tests to that playlist, it shows 2 tests - but only one is on the playlist. In above example in VS it is shown:
AreaCreateDuplicateRoot
AreaRetrieve_MustRetrieve