dotnet test <dll> --filter parameter is ignored when there are a large number of tests. All tests in all classes are run regardless of filter.

[fenjas]

• NUnit and NUnit3TestAdapter versions: 3.13.3 and 4.3.1
• Visual Studio edition and full version number (see Help About): VS 2022 17.4.2 (v 4.8.04084)
• What .net platform and version is being targeted: .NET 6.0

I have a VS C# solution with 3 test classes as per pic attached. I'm using a PowerShell script to execute these tests on a schedule using the following statements:

• dotnet test biditests.dll --filter TestCategory~Daily
• dotnet test biditests.dll --filter TestCategory~Monthly
• dotnet test biditests.dll --filter TestCategory~GroupBrandNames

The --filter parameter seems to be ignored, so when I run the dotnet command targeting ~Daily, all the tests in all 3 classes are executed instead of the ones targeted by the filter.

If I execute the dotnet command targeting ~GroupBrandNames or ~Monthly, the filter works fine presumably because there are only 2 and 60 tests respectively in each class. If I keep the no. of tests in the Daily class under 2000, everything works as expected. As of now it contains in excess of 3000 tests.

I also tried other ways to run the tests using the project instead of the the dll, and changing the filter to using the FQDN, etc. However, the issue is the number of tests.

I found an article which mentioned creating a .runsettings file and upping the AssemblySelectLimit from 2000 to something greater than the number of tests. I did this, however it does not seem to work (assuming I did it correctly, file attached) and from what I understood applies only when you're running tests from the IDE which is not my case.

As a fix of sorts, I ended up using a Environment.Exit() call in the OneTimeTearDown method after the last test fixture is executed so the runner exits before getting a chance to run tests from another class. It's ugly but it works for now.

Logging this here since I found a similar issue and thought mine would be related!

Thanks.