TestCaseSource error with dotnet test --filter, but works fine in VS and without filter

[ovebastiansen]

• NUnit: 3.12.0
• NUnit3TestAdapter: 3.16.0-dev-01208 and below
• Visual Studio edition and full version number: Community, 16.4-preview5
• A short repro: https://github.com/ovebastiansen/nunitadapterissue
• What .net platform and version: dotnet core 2.2

Running the code in Visual Studio works fine, Test explorer are sucsefully finding all test cases and are able to either run all or single. The problem arises either when running "dotnet test" or in Visual Studio Code with ".net Test explorer" extension. Then running with dotnet test --filter "FullyQualifiedName=TestFixtureSourceError.TestsBroken.Test1" failes with 0 Tests found.

I know there is a workaround, that in some cases work, with TestFixtureData, but why does it work in Visual Studio then? And also when using TestFixtureData everything is then broken in Visual Studio and in Visual Studio Code, the extension(that is not your concern) still can't find or start a test and not all dotnet test --filter works.

[ovebastiansen]

After more debugging I have found these two different cases dotnet test --filter "FullyQualifiedName=TestFixtureSourceError.TestsBroken(Windows Edgium).Test1" throws an exception from TfsTestCaseFilterExpression with the text Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.TestPlatformFormatException: 'Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.' But the command works perfect: dotnet test --filter "FullyQualifiedName=TestFixtureSourceError.TestsBroken(Windows Edgium).Test1"

Any pointers about where to look for a possible solution, if there are any?

I tried escaping the () in the FQN on discovery, but then everything broke. And on the way in, I think the error has happen before I am able to escape it.

[OsirisTerje]

@ovebastiansen The FullyQualifiedName has to follow the rules for a FQN for managed code. That means the name: TestFixtureSourceError.TestsBroken(Windows Edgium).Test1 can not work because it has a class with parameters. There is an RFC for FQN at Microsofts, https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0017-Managed-TestCase-Properties.md

You can reach out to them for comments on this. The MSTest adapter will implement this RFC first, then the 3rd party adapters will follow. But we try already now to follow this as far as we can.

[ovebastiansen]

Oh my bad, I notice a typo in my last comment The command that works is dotnet test --filter "FullyQualifiedName=TestFixtureSourceError.TestsBroken(Windows Edgium).Test1" So by escaping the () when creating the filter it works. I might take a stab at the extension to see if that has a entrypoint where I can escape the () and by that setup make my tests work. I will let you know of my progress

[OsirisTerje]

That is weird! I didn't expect that to work, but.... And, I can't really see the typo either, the two statements look equal to me..... I'm probably missing something but....

What is strange is that if you mean it works with command line dotnet, and not in VS, they are both using the adapter. If you set verbosity to 5 using the runsettings, you can check to see if they both call the same method, they should.....

[ovebastiansen]

There is no typo, just github is properly sanitizing statements ;) dotnet test --filter "FullyQualifiedName=TestFixtureSourceError.TestsBroken\(Windows Edgium\).Test1" Needed double \\ to get it to properly show what I mean

[ovebastiansen]

VS works out of the box currently, but dotnet with filter does not work dotnet test with no filter picks up three tests, but adding filter with the fqn needs \( to work I need to look into how VS does the filtering, if they escapes or not

[ovebastiansen]

https://github.com/nunit/nunit3-vs-adapter/blob/master/src/NUnitTestAdapter/NUnit3TestExecutor.cs The comments here are actually still valid ;) One method is called from the commandline and from tfs build and another is called when running from visual studio. The first has a list of dlls to test and a filter from the runcontext that breaks if there are () without any escapes, the commandline version From VS the method called has a list of tests to execute. I am not sure why this two are different

[OsirisTerje]

There are two RunTests methods in the interface, one with a list of sources (aka dll's or exe's), and the other with a list of test cases. The second is used from VS where it runs it's own discovery, and sends those tests further down. Thus the need for the RFC mentioned above in order to "think" the same about the names, otherwise they will not match.