dotnet-test-nunit icon indicating copy to clipboard operation
dotnet-test-nunit copied to clipboard

Playlists wrongly group tests

Open nepdev opened this issue 7 years ago • 2 comments

dotnet-test-nunit v3.4.0-beta-3, with NUnit 3 Test Adapter (on Visual Studio 2015 Update 3 Community Edition)

The project under test is ASP.Net Core targeting .Net 4.5.2 (i.e. not a "pure" dotnet project).

Creating a playlist with 20 tests (out of 100) by selecting in Test Explorer, right-click, Add To Playlist > New Playlist. Only these 20 tests show in Test Explorer. Now run "All Tests".

Result: The first 13 tests show in green, the next 7 tests are dimmed out (normally this indicates they were not run). At the bottom of the screen it says "18 tests passed". See screenshot nunit-error1

What should have happened: All 20 tests are highlighted green. At the bottom it says "20 tests passed".

Workaround: Do not use playlists. The default view with all tests, grouped by "outcome" works fine. See also other issue I submitted - it seems that the only grouping which works properly is the default "by outcome" view with "all tests".

Update Edit:

  1. I found that in addition to above the playlist file contains OTHER tests than what I had selected in the UI.

  2. And, reversely, the UI shows other tests than what the playlist contains. Example - Making a short playlist with two tests. File then, for example contains: <Playlist Version="1.0"><Add Test="StatsServicesTest.AreaTests.AreaRetrieve_MustRetrieve" /><Add Test="StatsServicesTest.AreaTests.AreaRetrieve_ProperData" /></Playlist> When filtering the tests to that playlist, it shows 2 tests - but only one is on the playlist. In above example in VS it is shown:

AreaCreateDuplicateRoot AreaRetrieve_MustRetrieve

nepdev avatar Mar 28 '17 13:03 nepdev