Test Adapter runs explicit tests when TFS TestCaseFilter is used

[Taron-art]

NUnit test adapter runs explicit tests when the TestCaseFilter is used. For example TestCategory!=Temp. I reproduced this issue using VSTest.Console and on the TFS Team Build 2012.

[Taron-art]

I looked on that more carefully and I see that it questionable bug: Explicit tests should be run when the category is explicitly selected. However, it is not consistent with the NUnit GUI and command line, because if you exclude the category from the test run, it will not automatically run all explicit tests from other categories\without category. The only workaround which I see is to include explicit tests in some additional category (e.g. Explicit). Can this issue be fixed in the adapter or its some "feature" of the VS test runner?

[CharliePoole]

NUnit itself handles an exclusion filter (one with not at the top level) specially. As you have noted it doesn't consider explicit tests to have been explicitly selected just because they are not excluded.

When you use a filter under vsconsole or tfs, we do create an NUnit filter but it's never an exclusion filter - it's just a list of the test names selected by VS. This is necessary because VS can filter on traits that are not recognized by NUnit and have no corresponding NUnit filter.

I'll leave it up to @OsirisTerje to make the final decision as to closing this issue, but I'm pretty sure it's something we can't fix, at least given the current version of NUnit.

[bussnik]

We also stumbled over this issue in our first use of TFS with the following scenario: When TFS runs the unit tests without TestCaseFilter, then the tests marked [Explicit] are not executed.

Then we had to exclude some other tests, so we tagged them with [Category("Local")] and set TestCaseFilter to TestCategory!=Local. As effect, the [Explicit] now were executed.

Our workaround is to tag all these tests with [Explicit, Category("Explicit")] and to set the TestCaseFilter to TestCategory!=Local & TestCategory!=Explicit. But this is somehow redundant and error-prone.

Maybe it is possible to either

• let the Explicit attribute be some subclass of the Category attribute named "Explicit", or
• let the NUnit (not the Visual Studio) TestRunner treat Category("Explicit") same as Explicit, or
• you have another solution more helpful than just closing this issue?

[CharliePoole]

There are actually two issues here:

1. How NUnit handles explicit. By design, NUnit ignores Explicit tests not directly selected. Directly selected means either by name or by inclusion through a category. Excluding a category does not cause the explicit test to be run. This works pretty well in the gui and console runners and in the VS test window using the adapter. Of course, it's not possible to exclude categories in the IDE, other than by not selecting them when other categories are selected.
2. How the adapter handles TFS filters. We do not translate TFS filters into NUnit filters. If we did that, this problem would not exist, but doing so would be a significant effort and would require updating any time the content of a TFS filter was enhanced by Microsoft. What we do is ask the filter to give us a list of tests. That list is used to create an NUnit name filter and we run all the tests that are passed to us by NUnit.

Although it has not been clear to me up to now, I think the real meat of this issue is in point 2. I'm changing the title to reflect that it's really about TFS filters and not NUnit filters. I think that's what the OP actually intended anyway.

We can fix this in one two ways:

1. Translate the TFS filter to an NUnit Filter.
2. Review the list of test names received from the filter after they are discovered by NUnit and remove any that are marked explicit.

Both of these will be a bit of work. We're about to do a review of outstanding issues in the V2.0 test adapter to decide which ones to fix in the next 2.x release and which ones to postpone to 3.0. Some may be fixed in both places. For the moment, I'm thinking this has to be in 3.0 but might not get fixed in 2.x. It depends on whether the fix we choose has dependencies on NUnit 3.0 features.

[CharliePoole]

To be handled in 3.0 by nunit/nunit3-vs-adapter#47

[chm-tm]

The work around mentioned by @bussnik also works in the Visual Studio IDE. You have to enter -Trait:"Explicit" -Trait:"Local" into the Test Explorer search bar.

[OsirisTerje]

@chm-tm @bussnik @Taron-art We're continuing this issue wrt having this resolved now in the NUnit3 adapter. See https://github.com/nunit/nunit3-vs-adapter/issues/47

Appreciate comments. :-)

[OsirisTerje]

Based on the discussion in nunit/nunit3-vs-adapter#47 (see this for further explanations) we'll reopen this and change the default behavior to not run explicit tests, unless explicitly stated.