vstest.console's `--TestCaseFilter` is ignored when listing tests

[julealgon]

When running the -lt or --ListTests command line parameter on vstest.console, the NUnit adapter will completely ignore the filter specified on --TestCaseFilter. This seems to go against the expected behavior, since for MSTests, the filter is properly considered for both running and listing tests.

---

With this very simple test class, we can already see the problem:

    [TestFixture]
    public class NunitTests
    {
        [Test]
        [Category("NunitCategory1")]
        public void NunitMethod1()
        {
        }

        [Test]
        [Category("NunitCategory2")]
        public void NunitMethod2()
        {
        }
    }

Here is the output for running a test using a filter. This works correctly and only a single test is run (NunitMethod1):

vstest.console "UnitTestFilteringSample.dll" --TestCaseFilter:"(TestCategory=NunitCategory1)" Microsoft (R) Test Execution Command Line Tool Version 15.5.0 Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait... NUnit Adapter 3.9.0.0: Test execution started Running all tests in C:\GithubJuliano\NUnitListTestsIssue\UnitTestFilteringSample\bin\Debug\UnitTestFilteringSample.dll NUnit3TestExecutor converted 2 of 2 NUnit test cases NUnit Adapter 3.9.0.0: Test execution complete Passed NunitMethod1

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0. Test Run Successful. Test execution time: 0.9984 Seconds

But this is the result from the --ListTests functionality, using the same filter:

vstest.console "UnitTestFilteringSample.dll" --TestCaseFilter:"(TestCategory=NunitCategory1)" -lt Microsoft (R) Test Execution Command Line Tool Version 15.5.0 Copyright (c) Microsoft Corporation. All rights reserved.

The following Tests are available: NUnit Adapter 3.9.0.0: Test discovery starting NUnit Adapter 3.9.0.0: Test discovery complete NunitMethod1 NunitMethod2

Notice how both NunitMethod1 and NunitMethod2 are found, even though NunitMethod2 does not have the "NunitCategory1" defined.

From my tests using a sample with mixed NUnit and MSTest tests, the --TestCaseFilter parameter seems to always produce a list with all NUnit tests, even if they fail the criteria. When filtering on top of MSTest tests however, we always see consistent results.

• NUnit v3.9.0
• NUnit3TestAdapter v3.9.0
• Visual Studio Enterprise 2017 v15.5.4
• vstest.console v15.5.0
• Project sample here: https://github.com/julealgon/NUnitListTestsIssue

[CharliePoole]

I can confirm that this is how the adapter is written. There is never any filtering applied on test discovery.

This makes sense given that it was written originally with the IDE in mind.

To make it work differently we would need to discover how the calls to the discovery entry differ when used in this way - probably by experimentation.

[julealgon]

I assume that the UI would never pass this parameter to you @CharliePoole , isn't that the case? If so, it would just be a matter of start to handle it and it would work both on CLI and inside VS.

[CharliePoole]

Precisely. We would need to find out what parameter is being passed and how. Our relation to the MS code is that we find out how it works by running experiments. 😸

[CharliePoole]

@jnm2 That's a little tacky to point out! 😸

The "initial upload" wasn't that initial. It was just copying the code that was already present from years back into a new repository. It's possible that the code was needed in it's old context and I'm asking whether it gets any use now, in its present context.

But I'd think it's a valid question even if the original code had been written by some earlier and dumber version of myself.