dotnet-test-nunit icon indicating copy to clipboard operation
dotnet-test-nunit copied to clipboard

Published 20 hours ago • verified publisher icon nunit

Playlists wrongly group tests

Open nepdev opened this issue 7 years ago • 2 comments

dotnet-test-nunit v3.4.0-beta-3, with NUnit 3 Test Adapter (on Visual Studio 2015 Update 3 Community Edition)

The project under test is ASP.Net Core targeting .Net 4.5.2 (i.e. not a "pure" dotnet project).

Creating a playlist with 20 tests (out of 100) by selecting in Test Explorer, right-click, Add To Playlist > New Playlist. Only these 20 tests show in Test Explorer. Now run "All Tests".

Result: The first 13 tests show in green, the next 7 tests are dimmed out (normally this indicates they were not run). At the bottom of the screen it says "18 tests passed". See screenshot nunit-error1

What should have happened: All 20 tests are highlighted green. At the bottom it says "20 tests passed".

Workaround: Do not use playlists. The default view with all tests, grouped by "outcome" works fine. See also other issue I submitted - it seems that the only grouping which works properly is the default "by outcome" view with "all tests".

Update Edit:

  1. I found that in addition to above the playlist file contains OTHER tests than what I had selected in the UI.

  2. And, reversely, the UI shows other tests than what the playlist contains. Example - Making a short playlist with two tests. File then, for example contains: <Playlist Version="1.0"><Add Test="StatsServicesTest.AreaTests.AreaRetrieve_MustRetrieve" /><Add Test="StatsServicesTest.AreaTests.AreaRetrieve_ProperData" /></Playlist> When filtering the tests to that playlist, it shows 2 tests - but only one is on the playlist. In above example in VS it is shown:

AreaCreateDuplicateRoot AreaRetrieve_MustRetrieve

nepdev avatar Mar 28 '17 13:03 nepdev

Update: tested the same issue on a "regular" desktop application project targeting .Net 4.5.2 (Windows Form application). The issue above does not occur in this project. It may possibly be related to the fact that the project under test in this issue is not a pure dotnet project but an ASP.Net Core project targeting the .Net 4.5.2 runtime.

nepdev avatar Mar 31 '17 19:03 nepdev

@nepdev Thanks for checking! @rprouse Looks like this is particular to dotnet-test-nunit then.

CharliePoole avatar Mar 31 '17 21:03 CharliePoole