VS Test Explorer does not show results for test cases that have custom names

[rjgotten]

Given a unit test that makes use of a sequence of test cases with custom name using TestCaseData.SetName, VS Test Explorer seems to fail to pick up the test result for any case which falls outside of the default {m}{a} test case naming pattern.

The Tests category in the VS Output panel will list:

No test matches the given testcase filter `FullyQualifiedName=Namespace.Class.Method`

where the FQN matches the test name affected.

Either NUnit or Visual Studio itself is doing some over-aggressive filtering here, most likely based on a full-string match using a regular expression, instead of a more conservative prefix match.

NUnit : v 3.11.0 NUnit3TestAdapter : v 3.12.0

---

This is bothersome if we want to write a custom test name for cases to also list the expected result. Right now, I'm working around this by faking the result as an additional parameter in the parameter list. This gets it passed whatever is applying the filter:

public static IEnumerable Cases
{
  get
  {
    return Enumerate()
      .Select(testCase => testCase
        .SetName($"{{m}}({{0}},{{1}},\"{testCase.ExpectedResult}\")")
      );

    IEnumerable<TestCaseData> Enumerate()
    {
      yield return new TestCaseData( /* ... */ ).Returns( /* ... */ );
      // ...
    }
  }
}

However, this is far from ideal, as we have to specify the arguments explicitly by index this way, and cannot use the {a} shorthand.

[OsirisTerje]

Hi! Could you add the code above into a repro project and upload that? Makes it way more easy to test out in every scenario.
I believe you are into something here, but I thought we had this working.

[rjgotten]

@OsirisTerje I'll try to find some time to hammer out a repro for this. I'll have to see if I can squeeze in some time for it somewhere this week, as I'm quite busy with high-priority other things.

[OsirisTerje]

@rjgotten Would be awesome if you could find time to create that repro project.

[rjgotten]

@OsirisTerje

Took the time to get you that repro: https://github.com/NetMatch/nunit-adapter-testcase

[OsirisTerje]

@rjgotten Thanks for the repro, made it so much simpler. Awesome explanation too!

I've copied in your text here, for comments:

1. Choose 'Run All' from the VS Test Explorer to run all tests
   • OK : All custom-named test cases show up as having run.
   • BUG : Original test method names remain present as not run.

=> This is a known bug with the Source Based Discovery, same as #613 , and I have reported it to the PG. Note: If you turn off source based discovery, it will disappear.

2. Right-click the 'Tests' node from the VS Test Explorer and choose to run only the tests inside that test fixture.
   • BUG : None of the test cases show up as having run.
     The VS Output panel mentions a warning in the Tests category:
     No test matches the given testcase filter `FullyQualifiedName=Testcase.Tests.Tests.Concat|FullyQualifiedName=Testcase.Tests.Tests.Add`
   • BUG : Original test method names remain present as not run.

=> If I start from blank, then I can repro this. If I use a Run All first, then it runs as it should the next time. This should trigger the NUnit discovery, so I'll check this further.

3. Right-click the 'Add' node from the VS Test Explorer and choose to run only that test.
   • BUG : None of the Add test cases show up as having run.
     The VS Output panel mentions a warning in the Tests category:
     No test matches the given testcase filter `FullyQualifiedName=Testcase.Tests.Tests.Add`
   • BUG : Original test method names remain present as not run.

=> Same as (2), repro.

4. Right-click the 'Concat' node from the VS Test Explorer and choose to run only that test.
   • BUG : None of the Concat test cases show up as having run.
     The VS Output panel mentions a warning in the Tests category:
     No test matches the given testcase filter `FullyQualifiedName=Testcase.Tests.Tests.Concat`
   • BUG : Original test method names remain present as not run.

=> Same as (2), repro.

NOTE: These errors are all related to Source Based Discovery, if that is turned off, it works as it should.

[johnmwright]

This issue has the same underlying cause as #622 and is resolved by PR https://github.com/nunit/nunit3-vs-adapter/pull/668

[OsirisTerje]

Checked with 3.16.0-dev, and now works:

Before:

After:

[OsirisTerje]

@rjgotten Pre-release package with fix: https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter/3.16.0-dev-01202

[rjgotten]

@OsirisTerje Thanks. I think the old project that spawned this issue is still around and kicking, so I'll give this update a try come next week.

[OsirisTerje]

THis issue is not fixed in 3.17 and above. The fix in 3.16.0 caused regression issues and was revoked, although the fix can be enabled using the options, see release notes and tips and tricks.