Visual Studio test runner behavior during execution

[D0tNet4Fun]

Consider the following test method:

[Theory]
[InlineData(1)]
[InlineData(2)]
public void MyTest(int data)
{
  Thread.Sleep(2000);
  if (data % 2 ==0) throw new Exception("Even numbers are not supported");
}

With theory pre-enumeration turned off, the VS test runner discovers only the test method MyTest. When the test method is run, it adds 2 tests for the test method: 1 passed and 1 failed. The test for MyTest shows as failed. Everything works as expected in the end, but not during the execution. For 2 seconds, MyTest shows as if passed and then it fails.

From what I can tell, the messages of interest that are queued on the message bus are:

1. test case starting
2. test starting
3. test passed
4. test finished
5. test starting
6. test failed
7. test finished
8. test case finished

For this sequence, the status of MyTest in the VS test runner changes to:

• In progress, after 1. ()
• Passed, after 3. Test Passed - MyTest(data: 1) shows in the test details.
• Failed, after 6. Test Failed - MyTest(data: 2) shows in the test details.

I think it makes more sense for the VS test runner to have the following behavior:

• change status to In Progress, after 1.
• show Test Passed - MyTest(data: 1) in the test details, after 4
• show Test Failed - MyTest(data: 2) in the test details, after 7.
• change status to Failed, after 8.

In short, the status should not be changed to Passed or Failed as long as the test case is running.

Disclaimer: I need this behavior for my test framework in which a test case is assigned a list of tests at run time - like a theory whose data is dynamic.

[bradwilson]

I'm not sure I agree that this is an issue. The test run is not complete, so temporarily marking something as passed (when it will later be failed) doesn't seem like that big of a deal.

[D0tNet4Fun]

The behavior I expect is subjective, based on my own understanding of the test messages. Yet it matches the behavior of the Resharper test runner for CLR. I guess it would be nice for these two test runners to behave in the same way. If this is by design though, then I can understand it is not an issue.

From the perspective of extensibility, I wish there was a way for me to customize the VS test runner in order to meet these requirements. I could deliver it in a separate package which is supposed to be picked up by Visual Studio instead of the default test runner. I'm not sure if this is possible or already available. It would be a nice alternative.