Add test case filter based on existence of a property

[tillig]

The documentation on TestCase filtering shows this example:

namespace MSTestNamespace
{
    using Microsoft.VisualStudio.TestTools.UnitTesting;

    [TestClass]
    public class UnitTestClass1
    {
        [Priority(2)]
        [TestMethod]
        public void TestMethod1()
        {
        }

        [TestCategory("CategoryA")]
        [Priority(3)]
        [TestMethod]
        public void TestMethod2()
        {
        }
    }
}

And it shows that you can run all the tests marked with [TestCategory("CategoryA")] using this command:

dotnet test --filter TestCategory=CategoryA

However, if there doesn't seem to be a way to run tests that are not marked with any test category.

Same for XUnit:

namespace XUnitNamespace
{
    public class TestClass1
    {
        [Fact]
        public void foo()
        {
        }

        [Trait("Category", "Nightly")]
        [Trait("Priority", "2")]
        [Fact]
        public void bar()
        {
        }
    }
}

I can run the test with a Category=Nightly... but I can't run choose to run tests with no Trait "Category".

I've tried these and failed:

dotnet test --filter Category='' dotnet test --filter Category=null dotnet test --filter Category dotnet test --filter !Category dotnet test --filter Category!~''

(Some of these come up with TestCaseFilter errors where they aren't considered valid expressions.)

My logic being the value of a non-existent property should be empty/null... but diving a little deeper into the filter code that doesn't seem to be the case. There doesn't seem to be a way to filter based on a property being/not being there.

I might propose one of the following solutions:

• Properties that don't exist might equal empty string, so Category='' would succeed.
• Add a $null value so Category=$null would succeed.
• Allow a property name without a value to indicate existence so !Category would work.

[singhsarab]

@tillig We haven't received any feedback for such a scenario before. I am just trying to understand more about the scenario we trying to enable. For example, I would usually give Category!=Nightly to achieve what you want instead.

@pvlakshm @cltshivash Can you please triage this. /cc: @genlu Please share your thoughts.

[tillig]

My specific use case is that I have a microservice project that has a fully automated CI/CD pipeline. About half of the tests, spread across multiple assemblies, are unit tests. The other half, also spread across multiple assemblies, are BDD tests that run against the service during various stages of deployment.

Right now, we have four types of tests:

• Unit - runs during the build
• Canary - sort of like BDD smoke tests, run against the service prior to taking any traffic
• PreRollout - canary taking traffic and running against an integrated suite of dependencies
• PostRollout - the canary version of the service is now the only one out there, make sure things are still working

The pipeline is evolving and more stages are being considered. As the service evolves and new unit test assemblies are added, it is easy for people to forget to add the [assembly: AssemblyTrait("Stage", "Unit")] attribute to the new assemblies.

Given the default type of test is effectively unit tests, we were hoping to be able to omit the trait entirely. The idea, then, being that a unit test is one that is not otherwise marked to be run in a particular stage of the deployment pipeline. Instead of this...

dotnet test Foo.csproj --filter Stage!=Canary&Stage!=PreRollout&Stage!=PostRollout

...we could do this:

dotnet test Foo.csproj --filter Stage=null

Without that, as new stages get added (we're considering, for example, a stage called "Security") we then have to also update all the build scripts and filters:

dotnet test Foo.csproj --filter Stage!=Canary&Stage!=PreRollout&Stage!=PostRollout&Stage!=Security

Being able to test for empty/null would allow a default filter to work, no extra test markings (or things folks can easily forget), and no need to update the filter as the deployment stages/pipeline evolve.