Revise `--minimum-expected-tests` behavior with Retry extension

[Youssef1313]

This is a hypothesis that I didn't test yet.

But I think if --minimum-expected-tests is used with Retry, it could be problematic.

Let's say a test project has 10 tests, and --minimum-expected-tests 5 is passed. If one test failed in the initial run, the subsequent run will run only the one failed test, causing --minimum-expected-tests to potentially fail the run?

We need to revise this scenario and try to support it.

At least if it already works, we will need a test to cover it.

[Evangelink]

I would hope this is handled but thanks for the issue and let's ensure we have some acceptance test for it!

[Youssef1313]

I just tested, seems not handled.

using System.Diagnostics;

namespace TestProject87;

[TestClass]
public sealed class Test1
{
    [TestMethod]
    public void TestMethod1()
    {
    }

    [TestMethod]
    public void TestMethod2()
    {
        // --results-directory for first run will contain "Retries" in the path and ends with "1". Only fail the first retry.
        if (Environment.GetCommandLineArgs().Any(arg => arg.Contains("Retries") && arg.EndsWith("1")))
        {
            Assert.Fail("Failing");
        }
    }
}

Running with:

dotnet run -- --minimum-expected-tests 2 --retry-failed-tests 1

Output:

MSTest v3.8.3 (UTC 17/03/2025) [win-x64 - .NET 8.0.16]
failed TestMethod2 (37ms)
  Assert.Fail failed. Failing
    at TestProject87.Test1.TestMethod2() in Test1.cs:19
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Test run summary: Failed! - bin\Debug\net8.0\TestProject87.dll (net8.0|x64)
  total: 2
  failed: 1
  succeeded: 1
  skipped: 0
  duration: 274ms


=====================
Tests suite failed, total failed tests: 1, exit code: 2, attempt: 1/2
=====================


Test run summary: Minimum expected tests policy violation, tests ran 1, minimum expected 2 - bin\Debug\net8.0\TestProject87.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 208ms

Test suite failed with and exit code different that 2 (failed tests). Failure related to an unexpected condition. Exit code '9'

If --minimum-expected-tests not set:

MSTest v3.8.3 (UTC 17/03/2025) [win-x64 - .NET 8.0.16]
failed TestMethod2 (53ms)
  Assert.Fail failed. Failing
    at TestProject87.Test1.TestMethod2() in Test1.cs:19
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Test run summary: Failed! - bin\Debug\net8.0\TestProject87.dll (net8.0|x64)
  total: 2
  failed: 1
  succeeded: 1
  skipped: 0
  duration: 499ms


=====================
Tests suite failed, total failed tests: 1, exit code: 2, attempt: 1/2
=====================


Test run summary: Passed! - bin\Debug\net8.0\TestProject87.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 192ms


=====================
Tests suite completed successfully in 2 attempts
=====================

[MarcoRossignoli]

I don't think they work together.

We need also to take into consideration that extension at that level(orchestrators) could not work always correctly or "make core feature more coarse" because users has to stay aligned to every new "core" change.

[Evangelink]

I think the min tests feature should self-disables itself (or lower down to 1 for attempts after 1st) when not on first attempt. I would expect a similar behavior when we implement test sharding.