Calculated worker setting

[JustusGreiberORGADATA]

Summary

It would be really cool if the number of workers could be the result of some simple division depending on the maximum CPU threads of the machine.

<MSTest>
    <Parallelize>
        <Workers>{Environment.ProcessorCount}/2</Workers>
        <Scope>MethodLevel</Scope>
    </Parallelize>
</MSTest>

Background and Motivation

We have some tests, that spawn a bunch of threads that we try to parallelize at the method level. For this we are configuring the following in our runsettings file:

<MSTest>
    <Parallelize>
        <Workers>0</Workers> <!-- max cpu threads of the system -->
        <Scope>MethodLevel</Scope>
    </Parallelize>
</MSTest>

But because each test method starts a bunch of threads this isn't really the right number to get good performance. Instead we have the possibility to set a static value like <Workers>4</Workers> but this is not that flexible, if we use many different machines.

It would be perfect if we could say "use half/one fourth of the available CPU threads as the amount of workers."

Proposed Feature

Allow simple calculations (maybe only division) to set the worker amount.

Alternative Designs

• C# Hook that can modify the workers before the tests run (Does not seem to be how testfx does configuration anywhere else.)
• Extra runsettings property <WorkersScalingFactor>0.5</WorkersScalingFactor> is weird if the worker setting is an absolute number.

[Evangelink]

Hi @JustusGreiberORGADATA,

Thanks for the feature suggestion. Could you please specify what you min by "maximum number of threads on the system"?

[JustusGreiberORGADATA]

Hi @Evangelink,

physical CPU cores + hyper-threading "cores". If you look at Intel or AMD datasheets this is what they call total number of threads:

https://www.intel.com/content/www/us/en/products/sku/230580/intel-core-i513500-processor-24m-cache-up-to-4-80-ghz/specifications.html

https://www.amd.com/en/product/8456

It is my understanding that if I specify <Workers>0</Workers> this is also the number of test workers the test platform will dispatch. But maybe I'm wrong about that 🤔.

[Evangelink]

When you set 0 we are currently limiting to the number set for Environment.ProcessorCount:https://github.com/microsoft/testfx/blob/1223683d9b4ee9e2dd784f8315b45a50698281a0/src/Adapter/MSTest.TestAdapter/Execution/TestAssemblySettingsProvider.cs#L44

[JustusGreiberORGADATA]

https://learn.microsoft.com/en-us/dotnet/api/system.environment.processorcount?view=net-7.0

The number of logical processors on the machine. If the process is running with CPU affinity, the number of processors that the process is affinitized to. If the process is running with a CPU utilization limit, the CPU utilization limit rounded up to the next whole number.

Yeah logical processors in normal consumer setups should be the same number as the CPU threads I am referring to.

The CPU affinity and utilization limit are interesting use-cases though. So I guess it would be better to update the proposal above to say:

<Workers>{Environment.ProcessorCount}/2</Workers>

Then it is 100 percent clear what I want out of this feature.

[Orphis]

On my machine with an AMD 5950X 16 cores / 32 threads, the value returned by Environment.ProcessorCount is 32.

Unfortunately, this is not optimal for running my computational intensive tests. Using Workers = 0 or 32, my 4000 tests can take a bit over 3m. Using Workers = 16, they take only 2m30s.

It would be great if it was possible to select the exact logical core count automatically rather than the thread count when running tests in parallel in some situations. Since the project is used by many other people with different environments, it's not an option to hardcode a specific value (though 0 is obviously much better in most situations).

[Evangelink]

It would be great if it was possible to select the exact logical core count automatically

@MarcoRossignoli are you aware of any API in .NET that would give us a logical count?

So I guess it would be better to update the proposal above to say:

<Workers>{Environment.ProcessorCount}/2</Workers> Then it is 100 percent clear what I want out of this feature.

I prefer to avoid adding too much logic inside the runsettings. I have too much experience of it starts with a few operators and end up as a fully fledge computation expression. For such case, I think we could call a method (register a callback) with MSTest runner that we would call inside MSTest so users could provide any arbitrary logic.

@MarcoRossignoli @nohwnd WDYT?

[nohwnd]

are you aware of any API in .NET that would give us a logical count?

Environment.ProcessorCount returns the logical count. That is a number of threads that can run at the same time. On a normal CPU with hyperthreading it is physical cores * 2. Usually this is about the right number of threads to start for CPU bound work. And we cannot figure out the exact right amount on behalf of the user.

If each test starts additional 3 threads, then they (probably) will get more throughput using logicalCores/3 (minus some that we use for vstest overhead, if this is vstest).

If they start a lot of IO work, and use separate blocking threads for it, then more throughput can be achieved by upping the number workers even above the CPU count (on the expense of memory).

This also assumes that they are running just 1 test process at the same time.

I prefer to avoid adding too much logic inside the runsettings.

I was considering adding this setting to VSTest as well so user can say total cores/2, or total cores-10%. But it seemed too easy to do from commandline via inline runsettings, since the Environment.ProcessorCount is ENV variable.

[JustusGreiberORGADATA]

But it seemed too easy to do from commandline via inline runsettings

But this would not work inside the test explorer in VisualStudio, would it? Because this would be my main use-case.

[nohwnd]

As a workaround you can subclass the ParallelizeAttribute and calculate the factor in code.

[assembly: ScaleParallelize(0.5, Scope = ExecutionScope.MethodLevel)]

public class ScaleParallelizeAttribute : ParallelizeAttribute
{
    public ScaleParallelizeAttribute(double scalingFactor)
    {
        var workers = (int) Math.Round(Environment.ProcessorCount * scalingFactor, 0, MidpointRounding.AwayFromZero);
        if (workers == 0)
        {
            workers = 1;
        }

        Workers = workers;
    }
}

Just a warning that I don't see anyone doing this (on https://grep.app), so it is not a common scenario and might break later, or not be fully compatible with our analyzers.