Class Cleanup gets called immediately after a single test case using VSTestConsoleWrapper

[agadekar]

Describe the bug

I am using VsTestConsoleWrapper api to run tests. I have added same test method twice using below code -

Collection<TestCase> cases = new Collection<TestCase>();
  cases.Add(testCase);
  cases.Add(testCase);
var  consoleWrapper = new VsTestConsoleWrapper(runnerLocation, new ConsoleParameters { LogFilePath = logFilePath });
consoleWrapper.RunTests(cases, defaultRunSettings, handler);

However the ClassCleanup method gets called immediately after the first time test method is called and does not wait for the second instance to finish. Is this expected?

Expected behavior

class initialize  started at 3/9/2023 7:19:28 PM
test initialize  started at 3/9/2023 7:19:28 PM
TestA started at 3/9/2023 7:19:28 PM
TestA ended at 3/9/2023 7:20:18 PM
test cleanup  started at 3/9/2023 7:20:18 PM
class cleanup  started at 3/9/2023 **7:21:09** PM
test initialize  started at 3/9/2023 7:20:19 PM
TestA started at 3/9/2023 7:20:19 PM
TestA ended at 3/9/2023 7:21:09 PM
test cleanup  started at 3/9/2023 7:21:09 PM

Actual behavior

Logs from Test Run

test initialize  started at 3/9/2023 7:19:28 PM
TestA started at 3/9/2023 7:19:28 PM
TestA ended at 3/9/2023 7:20:18 PM
test cleanup  started at 3/9/2023 7:20:18 PM
class cleanup  started at 3/9/2023 7:20:19 PM
test initialize  started at 3/9/2023 7:20:19 PM
TestA started at 3/9/2023 7:20:19 PM
TestA ended at 3/9/2023 7:21:09 PM
test cleanup  started at 3/9/2023 7:21:09 PM

Additional context

Note how Class Cleanup was called at 7.20.19 before second run of TestA finished (7.21.09). Shouldn't it be called after 7.21.09?

[agadekar]

Also, logs added using https://learn.microsoft.com/en-us/windows-hardware/drivers/taef/wexlogger do not get captured in TestResultMessage. Is there a way to do that?

[engyebrahim]

Hi @agadekar,

I tried this in small repo with mstest v3.0.2 and the `classcleanup' is called after the two methods.

so please can you check what is the mstest version in your project? and if it's the latest can you send sample project? or use --diag:log.txt and send the logs file to us? if the logs has sensitive data you can add a feedback ticket in VS and then modify it by sending the logs as privet

[agadekar]

SampleProject.zip

we are using 3.0.2v of mstest

[engyebrahim]

Thanks @agadekar,

in here the problem when happens when we run the same test twice from a class has only one test and that's because in the code we check if all tests run then call the class cleanup which happens after first call to the test.

[agadekar]

can we please add it as a feature to recognize duplicate test cases in a single run as individual test cases and run class cleanup only after all these test cases have finished running?

[nohwnd]

@engyebrahim @Evangelink I am not sure why this was moved from testfx for vstest, this looks like a problem of testfx rather than the platform.

The best solution I can think of on platform side is to select distinct tests based on IDs, but that will impose the behavior on all frameworks (admittedly we do similar thing with sources where we select unique sources).

[nohwnd]

@Youssef1313 this should probably be addressed on mstest side, do you see that as feasible? Rather asking here before we play another round of ping-pong with this issue.

[Youssef1313]

Moving to MSTest for now. I'll investigate.

[nohwnd]

I remember investigating this, the code that runs assembly initialize and assembly cleanup is imho using hashtable to keep the tests to run, but we provide 2 tests so after the first test it will remove the single instance, and run asssembly cleanup.

[Youssef1313]

This is likely already fixed (same fix/issue as https://github.com/microsoft/testfx/issues/3018). So I'm going to close this.