microsoft
Please execute data driven unit tests sequentially while debugging, not in parallel
Summary
Currently, debugging data driven unit tests is hell, because all of a data driven test method’s variations are run in parallel.
It’s impossible to debug a particular test scenario then.
Example
[TestMethod]
[DataRow("a")]
[DataRow("b")]
[DataRow("c")]
[DataRow("d")]
public void Test(string input)
{
...
}
Background and Motivation
When debugging a data driven unit test and hitting F10, F11 or F5 on a breakpoint, it's not the next instruction within the same text execution that the debugger halts. Instead, it's the next instruction from another data driven test within the same method.
Given there are multiple data rows, it's impossible to follow what's going on.
Proposed Feature
During debugging, data driven tests should never run in parallel.
/ref: https://developercommunity.visualstudio.com/t/MS-Test:-Please-execute-data-driven-unit/10841744
I don't think we should influence how execution works based on whether or not debugger is attached. You can either temporarily disable parallelization altogether, or freeze the other threads. See Freeze and thaw thread execution. I'm going to close as by-design.
That approach seems very cumbersome to take with every single debugging session.
May suggest to extend the ExecutionScope with a third attribute: None. This could then be set in the project file, independent for distinct configurations.
Setting parallelization options in project file is tracked by https://github.com/microsoft/testfx/issues/4116.
Having a new ExecutionScope for Parallelize attribute may be something to consider. I guess something like MethodLevelExcludingParameterizedTests or something like that where test methods are parallelized, but single test cases of the same method are not.
Meanwhile, I'm curious if maybe specifying FoldingStrategy = Fold for all the DataRows can have somewhat the same effect with regards to parallelization?
Or maybe what we should fix here is the ability to run single test case altogether. Re-opening to track the other suggestions. But I still think the original suggestion isn't something we should attempt to do.
Another option, when using MTP, would be to have an explicit program.cs and to have a pragma limiting the max parallelism to one when debugging.
@Youssef1313 this is what we did when we were using TATF.
Or maybe what we should fix here is the ability to run single test case altogether.
Yes that seems like a good problem to solve.
But I can also see having a button in the UI, for Debug tests (non-parallel) which would be more universal across running multiple tests. I would use that feature from time to time, so my normal run can take advantage of being as parallel as possible, and debugging can be non-parallel, because in some cases it does not matter how the tests run, or maybe I am not really sure if my bug in caused by the tests running in parallel or not.
From my perspective, the best fit solution may be this:
When tests are run and a debugger is attached, as soon as the debugger halts execution for the first time during a debugging session – either because an Exception was thrown or a break point was hit –, the runner should permanently suspend all other threads automatically. The other threads should not continue to run until the current thread has ended. At that point, parallelism should be enabled again until the debugger halts again. Then, when the debugger halts execution again, all other remaining threads should be suspended automatically, and so on …
Out of curiosity, because this issue is really bugging me: Is there any progress in regard to designing a solution? I think I provided a deliberate proposal above.