Execute explicit tests using the TestCaseFilter argument on command line

[jnm2]

Following on from #47, we decided in 3.10 to add the Explicit UI grouping header for test cases that are explicit or have an explicit ancestor, and we made the test case filter behave more consistently by preventing explicit tests from ever running.

For 3.11, we want to think through the TestCaseFilter language for a good way to enable running explicit tests from the command line. This will be primarily for quick interactive command line testing such as dotnet test --filter "X", not CI scenarios.

Possibility 1: TestCategory contains syntax

Possible syntaxes	Runs non-explicit	Runs explicit
(TestCategory not compared to Explicit)	✔	❌
… & TestCategory != Explicit	✔	❌
… & TestCategory = Explicit	❌	✔
… & (TestCategory = Explicit | TestCategory != Explicit)	✔	✔

There is an objection to TestCategory = Explicit because that would treat [Category("Explicit")] tests like [Explicit] tests. This could be a bug or a feature, depending on your point of view.

Possibility 2: Explicit property syntax

Possible syntaxes	Runs non-explicit	Runs explicit
(Explicit not compared)	✔	❌
… & Explicit = false	✔	❌
… & Explicit != true	✔	❌
… & Explicit = true	❌	✔
… & Explicit != false	❌	✔
… & (Explicit = true | Explicit = false) (+ 3 more permutations)	✔	✔
… & Explicit = any (might require custom parsing and custom evaluation)	✔	✔

This might be the most intuitive. These options need to be tested to make sure they are feasible—even if we do our own parsing, VSTest will still fail if its own parser doesn't like the user's TestCaseFilter. That's good because we want to stay as close as possible to the VSTest filtering language for interoperability.

Possibility 3: Explicit operator-free syntax

Regardless which of the syntaxes above we add, we might try adding Explicit as a synonym since VSTest allows a name to be mentioned without an operator. This might require custom parsing and custom evaluation. The benefit to this one is when you're at the command line and you want a minimal phrase to type, such as Explicit & TestCategory = LongRunning.

Possible syntaxes	Runs non-explicit	Runs explicit
… & Explicit	✔	❌

⚠ Normally this would run all tests whose FullyQualifiedName contains Explicit. This is inconsistent in the same way as treating Explicit as a category: --filter Explicit1 would match all tests whose names contain Explicit1, and Explici would match all tests whose names contain Explici, but we would override Explicit to match all explicit tests regardless of FUllyQualifiedName.

Possibility 4: FullyQualifiedName always selects explicit tests

Possible syntaxes	Runs non-explicit	Runs explicit
… | FullyQualifiedName = Namespace.Class.Method	✔	Namespace.Class.Method only

---

We can do any combination of the above four that appeals to us. Which ones do you like best? Is there an alternative syntax you'd like to see?

[jnm2]

I discovered that --filter SomeName, where SomeName is not compared via =, != or ~, means the same as FullyQualifiedName ~ SomeName (docs). Updated Possibility 3 above.

[CharliePoole]

@OsirisTerje I want to remind you of our off-line discussion. As part of that, I offered to provide code that would make the filtering work without any explicit mention of explicit 😄 , the way it is supposed to. You have not yet gotten back to me on our overall conversation, so I haven't done any work on the specific feature up to now.

[jnm2]

@CharliePoole If you have an alternative proposal, would you mind sharing it here?

[CharliePoole]

It has been shared and discussed for quite some time.

[jnm2]

I'm still trying to get a handle on that. Are you referring to options 2 or 3 here?

[CharliePoole]

@jnm2 I was on my phone earlier and couldn't find it. See #47, specifically my option #1, which I've been advocating for a long time. I don't see much use of further discussion, however, until I resolve privately with @OsirisTerje whether he wants me to contribute.

If I end up providing a PR, there should be plenty of opportunity for discussion. Meanwhile, I think it's pretty clearly spelled out in #47.

[jnm2]

Okay, as far as I can tell you're referring to this option 1:

1. Continue to use the MsTest filter language but parse it ourselves rather than leaving it to VS.

If I understand correctly, that's an implementation strategy. I don't see proposals of outcome specs though, which is what this issue is for. Before deciding the implementation approach I'd like to discuss the end result we want for CLI users. E.g. given syntax X, subset Y of all test cases are selected.

I've read #47 a few times now and I might be missing something, but just saying we're going to do it the NUnit way isn't sufficiently clear for me. Especially in light of the fact that any user or software interfacing with dotnet test and VSTest.Console is going to be selecting test cases (leaf nodes only), not NUnit hierarchical tests, since that is what VSTest and the VSTest filter syntax are about.

[CharliePoole]

@jnm2 What I'm saying is that when you select TestCategory=X using the command-line, it should automatically work the same as when you select in the console runner using cat==X. The user should not have to change their thinking about how categories work when switching between nunit3-console, dotnet test and vstest-console. Is that clearer?

BTW, this plan is completely orthogonal to what you are doing with an Explicit trait, AFAICS. Having an explicit trait would allow us to close #47 as wontfix rather than done.

[jnm2]

Good, so I wasn't far off. That is clearer, thanks. I think that is a good idea only if users do not have to change their thinking about how categories work when running MSTest and xUnit tests together with NUnit. Or when switching between different projects. I think the priority is on dotnet test --filter and VSTest.Console /TestCaseFilter: working predictably across adapters in their job of filtering test cases. I hope we do not prioritize making VSTest entry points (dotnet test/vstest and VSTest.Console) filter consistently with NUnit entry points (nunit3-console and dotnet nunit when we have it). Icing on the cake would be nice if it doesn't compromise the first priority.

[mscrivo]

I just ran across this issue and landed here. FWIW, I believe the current behavior is non-intuitive and possibility 4 above is the most intuitive option to me.

I have a fixture that I would like to mark as explicit, so that 1) developers do not run them by default in VS, 2) they are not run by default in CI and 3) can be run on demand by developers and 4) can be run in CI via specifying some kind filter/flag.

1,2 & 3 are all working as expected, but when I tried running dotnet test --filter "FullyQualifiedName~Namespace.Class" or dotnet test --filter Category=MyExplicitCategory, neither one of them worked. I understand that choosing the category probably shouldn't run the explicit tests, because you could have some explicit and some not in a category. But if I specify the class name, it should definitely run them.

[OsirisTerje]

@mscrivo Just specifying the class should not run test methods that are Explicit, but if the whole class is Explicit, it should be run. Is this what you see ?

Can you upload (or point to) a small repro project ?

Not sure I understand what you mean by "MyExplicitCategory". Can you explain ?

[mscrivo]

@OsirisTerje yes, I have [Explicit] on a class and I cannot run the class when using the FullyQualifiedName filter.

[OsirisTerje]

@mscrivo Thanks! That at least is a bug... or.. when reading above and trying to remember back, this issue has probably not been all implemented. The list above is a set of possible solutions. But we should be able to handle this for the next version.

[dutts]

Came across this issue when googling, am I right to assume that currently there is no way of running tests marked with [Explicit] from dotnet test?

[OsirisTerje]

@dutts With the adapter 3.17 (or lower), and with VS 2019 or VS 2017 later revisions, that is true.
But if you try to use the 4.0 alpha adapter it should work again. https://www.nuget.org/packages/NUnit3TestAdapter/4.0.0-alpha.1 (Note: It currently require latest NUnit 3.12)

[dutts]

@dutts With the adapter 3.17 (or lower), and with VS 2019 or VS 2017 later revisions, that is true. But if you try to use the 4.0 alpha adapter it should work again. https://www.nuget.org/packages/NUnit3TestAdapter/4.0.0-alpha.1 (Note: It currently require latest NUnit 3.12)

Thank you, just tried that and it works.

[OsirisTerje]

@dutts Nice! Note it is an alpha version, so if you find anything that doesn't seem right, we would appreciate issues on that :-)