AssemblyInitialize/Cleanup should not be executed when all tests are ignored

[gao-artur]

Describe the bug

When all tests in the project are ignored, the AssemblyInitialize method is not executed. The AssemblyCleanup is always executed. I'd expect both methods to always execute or not execute. The answer to this issue suggests the AssemblyCleanup will always execute, so I expect the same from AssemblyInitialize (otherwise, what the cleanup method supposed to clean?)

Steps To Reproduce

Set the breakpoint in AssemblyInitialize and AssemblyCleanup and debug

[TestClass]
public class UnitTest1
{
    [AssemblyInitialize]
    public static void AssemblyInitialize(TestContext context)
    {
    }

    [AssemblyCleanup]
    public static void AssemblyCleanup()
    {
    }

    [TestMethod, Ignore]
    public void TestMethod1()
    {
    }
}

Expected behavior

Both AssemblyInitialize and AssemblyCleanup are executed

Actual behavior

Only AssemblyCleanup is executed

AB#1850400

[engyebrahim]

Hi @gao-artur, I confirm that I'm able to reproduce the issue.

[Evangelink]

Hey @engyebrahim,

Could you check if that something that never worked (you can test with a few previous versions, e.g. 2.2.10, 2.2.8, 2.2.4) or if it's a regression we introduced in v3? If it's a regression we need to work on it soon.

[engyebrahim]

@Evangelink I can repro in these versions 2.2.10, 2.2.8, 2.2.4.

[Evangelink]

@gao-artur if both methods were skipped, would it be causing any issue for you? Looking at the issue and at the code, it seems more logical to me that they are both skipped if there is no test to run (no test or all tests are skipped). Because the cleanup is currently called, I would need to postpone the fix to v4 so I am wondering if there is any bad impact on your side.

[gao-artur]

Skipping both is even better. I see no point in initializing something that won't be used, especially if it's a heavy operation like creating and initializing DB, for example.

[Evangelink]

Postponing to v4 then as it is a breaking change (I agree that current behavior is not right).

[nohwnd]

I agree that setups and teardowns should run only when at least 1 test will run from the assembly, otherwise we are just wasting the effort.

...AssemblyCleanup will always execute, so I expect the same from AssemblyInitialize (otherwise, what the cleanup method supposed to clean?)

In my opinion AssemblyCleanup should not rely on AssemblyInitialize. Assembly initialize can throw an exception, be cancelled, or aborted, or be empty. Or the things that need cleaning up were created by tests, and not by assembly initialize.

So ideally Assembly Cleanup waits for AssemblyInitialize to finish (either successfully or unsuccessfully), but otherwise is not tied to it.