MaxCpuCount run settings not honored by xunit vstest test adapter (test are run in parallel by default)

[pkrukp]

Hi,

it seems that xunit vstest test adapter by default runs tests in parallel.

There is a vstest run settings MaxCpuCount (https://github.com/Microsoft/vstest-docs/blob/master/docs/configure.md) which should control this. The default value is 1 which means tests should not be run in parallel. Changing it does not affect xunit.

As a workaround, I found https://github.com/xunit/xunit/issues/856#issuecomment-353802785 which says you can use DisableParallelization setting - it does work, tests are not run in parallel if it's set to true.

But I think it'd be nice if xunit test adapter respected this setting, and worked consistently with test adapters for other frameworks (do not run tests in parallel by default).

[bradwilson]

FWIW, using our attribute (or configuration element) is the only way to ensure parallelization is turned off in all runners, not just the VSTest adapter.

[Mrnikbobjeff]

Okay I hope this is not gravebumping. I had this problem some months ago and it took me a while to realize what @bradwilson said. I debugged the vs unit test runner and found out that if you disable parallelization in the ui you simply dont have a maxcpucount setting. As we dont want to introduce a breaking change and have all unit tests run on a single thread when they do not have a maxcpucount attribute and instead rely on the disableparallelization attribute I would suggest checking whether we are in design mode and have no MaxCpuCount attribute, or one with the explicit value of 1. If so, we should disable parallelization. If paralellization is enabled in the test runner, the MaxCpuCount is set to 0, so we would simply not set disableParallelization if this element exists. I have a local working branch of this including unit tests and everything, I was wondering whether this would be acceptable as a pr.

[bradwilson]

A quick look at our source shows that we do not read MaxCpuCount.

[bradwilson]

We are not currently accepting any PRs, because the 2.x code line is dead, and it's not clear how much of the existing code will be used in 3.0. There's no point in wasting time now on a PR for what may be ultimately dead code.

[clairernovotny]

Revisiting this, the VS Adapter will still be needed in 3.0. I'd accept a PR that read that value.

[bradwilson]

The old documentation site that was originally linked here had a very terse description of its purpose:

Degree of parallelization, spawns n test hosts to run tests. Default: 1. Max: Number of cpu cores.

Reading the updated documentation is interesting:

This setting controls the level of parallelism on process-level. Use 0 to enable the maximum process-level parallelism.

This setting determines the maximum number of test DLLs, or other test containers that can run in parallel. Each DLL runs in its own testhost process, and is isolated on the process level from the tests in other test DLLs. This setting doesn't force tests in each test DLL to run in parallel. Controlling the parallel execution within a DLL (on the thread-level) is up to the test framework such as MSTest, XUnit or NUnit.

The default value is 1, meaning that only one testhost runs at the same time. A special value 0 allows as many testhosts as you have logical processors (for example, 6, for a computer with 6 physical cores without multi-threading, or 12, for a computer with six physical cores with multi-threading).

The number of distinct DLLs in the run determines the actual number of testhosts started.

My interpretation of this is that there is nothing for us to do here. This appears to be a configuration value that VSTest itself uses to determine how many instances of the test host to spawn (one per test assembly) to run assemblies in parallel.

More to the point, everything listed in that RunConfiguration section appears to be values that are read and interpreted by VSTest, and not by the test adapters.

(The closest analog we have is the parallelizeAssembly configuration file element, but it's not exactly a match here. We use this with our runners to determine if/when to run assemblies in parallel, but there's no way today to specify a limitation of how many can be run in parallel at once.)

As such, I'm going to close this issue.