nunit3-vs-adapter
nunit3-vs-adapter copied to clipboard
nunit
RunAll does not run all tests with Assembly Parallelizable(ParallelScope.All)
Using NUnit 3.9 and adapter 3.9, I added [assembly: Parallelizable(ParallelScope.All)] to a project that I use for testing NUnit issues to test running everything in parallel. When I use Run All, some tests are either not run, or not reported. The tests that don't run vary between runs and occasionally all tests run. Looks like a race condition.
I ran the same tests from the command line and all tests ran every time.
Without looking at the code yet, my guess is that the test start and test end events are getting interleaved causing them to not get reported? Does the adapter expect test start then test end for each test?
I cannot reproduce while debugging the tests.
The test project is at https://github.com/rprouse/nunit-tests, The Parallelizable attribute is commented out, but is in the nunit-v3 project in AssemblyInfo.cs if anyone wants to test.
I am using Visual Studio 2017 15.5.6
It's hard to imagine that this can be anything other than a reporting problem, since NUnit doesn't know or care it's being run by the adapter. One way to double check would be for the adapter to get the final result after run all and report how many tests were run. At present, the adapter ignores the final result and only deals with the individual events sent from NUnit.
I am almost positive that it is a reporting error. If I debug tests instead of run them, the breakpoints are always hit.
I tested it with 15.5.6, and got tests that were shown as not run. I then enabled a runsettings file with the dump setting, and got the file out. Checking a few of the tests that are shown as not run, shows as having been run and passed in the dump file. I've enclosed the dump file.
Since the adapter registers the events and sends them to the host at the same time as it writes the dump file, I find it unlikely this is any bug in the adapter. It more seems like a bug in the VSTest host.
