Filter for tests that are uncategorized

[Youssef1313]

Discussed in https://github.com/microsoft/testfx/discussions/5134

^{Originally posted by mscottford February 25, 2025} I'd like the ability to generate a list of tests that don't have a category specified. My use case is that I'm trying to monitor the number of tests in my test suite that have not been categorized so that I can ensure the number continues to go down over time as the team adds categories to tests.

I found https://github.com/microsoft/vstest/issues/2818 which concluded with a note that such a feature wouldn't be implemented in vstest, so I thought I'd bring up the idea here.

I took a peek at the source for --treenode-filter to get a sense about whether or not it would work. The only thing that isn't clear to me from reading through it is how categories would be represented for tests that haven't had a category defined. Would that be an empty string? And then I could write an expression that searches for tests with a 0 length category? Perhaps with ^$.

[nohwnd]

Yeah I guess figuring out the most intuitive way to represent null is the hardest thing here. It should also survive easily being provided on command line to a child process, and being combined with other categories. E.g. run all tests that are non categorized, or have short run time:

category == short || category == ???

[nohwnd]

Our proposal here is to simply leave the value empty, such as testcategory=

This has some edge cases regarding quotes and spaces.

for --filter

The current implementation already seems to be cutting trailing whitespace between = and the end, or the next operator.

testcategory=aaa , testcategory= aaa will match [TestCategory("aaa")] in both cases.

testcategory= now fails, so while [TestCategory("")] and [TestCategory(" ")] [TestCategory(null)] are all possible to define, they are unmatchable by the current filter. After the change only [TestCategory(" ")] will be unmatchable, because of the whitespace trimming. I think this is okay, as we would not recommend anyone to use whitespace only based categories.

" is used as literal, so as long as the user properly escapes the value, it can be used in the filter, if you specify testcategory=`"aaa`" it will match [TestCategory("\"aaa\"")]

This makes it impossible for us to handle cases where the category is surrounded by whitespace, or is just whitespace [TestCategory("aaa ")]` because we cannot provide the value including the whitespace, because of the trimming and adding quotes will include the quotes in the literal. This is also okay to me, because the current filter already has no way of matching those cases either.

This would of course apply to all the operators and possible categories. Special care needs to be taken around the implicit filter, that is defined as FullyQualifiedName~theprovidedstring, because there would empty string to semantically mean "no filter" rather than "all methods that have empty fqn", but providing empty string to --filter already throws "Unhandled exception. System.ArgumentNullException: Value cannot be null. (Parameter 'filterString')" which is ideal.

We might consider which filters do and do not support empty value, because for example for the FullyQualifiedName it probably does not make sense to accept empty value, and it might prevent the user from providing an incorrect filter.

for --tree-node-filter

TBD

[nohwnd]

Adding to 3.10 to keep it on our radar, but design needs to be fiinalized first.

[cbersch]

I would also need such a filter.

My use case is for NUnit to run all tests which don't have a category and some explicit tests with a certain category. NUnit doesn't allow selection of Explicit tests if a not filter is included:

[Test]
public void PleaseExecute() { }

[Test, Explicit, Category("MyExplicit")]
public void PleaseExecuteExplicit() { }

[Test, Category("DoNotExecute")]
public void DoNotExecute() { }

Using

dotnet test --project MyProject.csproj --filter "TestCategory != DoNotExecute | TestCategory = MyExplicit"

doesn't work, because of the not filter. So having the possibility to use

dotnet test --project MyProject.csproj --filter "TestCategory=MyExplicit | TestCategory="

would be nice.

[Evangelink]

I find the TestCategory= syntax quite confusing and I would probably feel like something was stripped during parsing. Could we instead introduce a new element to the filter like NoTestCategory or something along these lines?

Also, I don't see it documented but I am pretty sure that the TestProperty attribute can also be used in filters and would require a similar design.

[nohwnd]

Yeah that is one disadvantage, I agree.

Advantage of just = is that it is literally no value, so there is no chance that this conflicts with existing user identifiers, and that it is universal - user does not have to remember the exact identifier for test category, or property. And the user don't have to remember which version of "nothing" we used for the identifier: NullTestCategory / EmptyTestCategory / NilTestCategory / NoneTestCategory.

edit: Also the NoTestCategory, that was in the example, that I've read the whole time as None.

[nohwnd]

Probably the most reasonable thing would be to make it None, and warn / fail on tests that use that category / property / whatever else? After all the main problem is to not conflict with existing thing and we won't be able to reasonably do that unless we literally use the "nothing" approach TestCategory=.

[nohwnd]

^^^ @Youssef1313 @Evangelink any opinions on simply using None?

[Youssef1313]

I'm personally fine with TestCategory= if there are no ambiguities with parsing.

What I'm failing to understand is why we are ever needing to make any changes in testfx for that? Shouldn't that be changed in vstest?

[Evangelink]

^^^ @Youssef1313 @Evangelink any opinions on simply using None?

Fine by me! Let's introduce an analyzer in MSTest to guard users against using None

[Youssef1313]

I'm not opposed to that as well. What I would like to understand is:

1. What parts related to filter we call VSTest directly for, and what parts we "copied" from vstest.
2. What parts need to be modified.
3. For anything we copied from vstest, I would like to get it done there first.

[Evangelink]

Needs to be double checked but I recall we copied the VSTest logic for filtering only for the case of MTP, for VSTest we still rely on VSTest logic. But this might have changed when we added support for TestProperty (kvp).