Document default parallelizability of STA tests

[jnm2]

Things I couldn't figure out by reading https://docs.nunit.org/articles/nunit/writing-tests/attributes/parallelizable.html and https://docs.nunit.org/articles/nunit/writing-tests/attributes/apartment.html:

• Can any STA test run in parallel with any other STA test, and does this happen by default?
• Can any STA test run in parallel with any MTA test, and does this happen by default?

MTA tests running in parallel with each other by default seems obvious, but it could be stated too.

Also surprising to me is NUnit creating five STA threads. At one point in the past, I had thought there was only one STA queue. I would be interested in knowing if we could document the queues that NUnit uses by default.

[OsirisTerje]

Agree, we need to get this properly documented. It is however many variables involved, including which runner is being used and the different settings involved.

As far as I have understood it:

For fixtures:

Can any STA test run in parallel with any other STA test, and does this happen by default?

All tests in a fixture marked as an STA, run sequentially. Fixtures marked as an STA can run in parallel with other STA fixtures.

Can any STA test run in parallel with any MTA test, and does this happen by default?

I am pretty sure it can, as an STA fixture can run in parallel with any test in a MTA fixture.

For tests:

A test marked as STA will run in its own thread. Multiple tests marked as such should be able to run in parallel with each other, each in their own thread.

[jnm2]

Interesting. These thread IDs start coming out the same if ParallelScope.Children is used, unless you are debugging:

using NUnit.Framework;
using System.Threading;

[assembly: Parallelizable(ParallelScope.Children)]

public class Fixture
{
    [Test, Apartment(ApartmentState.STA)]
    public void Test1()
    {
        TestContext.WriteLine(Thread.CurrentThread.ManagedThreadId);
    }

    [Test, Apartment(ApartmentState.STA)]
    public void Test2()
    {
        TestContext.WriteLine(Thread.CurrentThread.ManagedThreadId);
    }
}

[jnm2]

Fixtures marked as an STA can run in parallel with other STA fixtures.

I was not able to reproduce this. The thread ID is the same for me, so the tests can't be running in parallel:

using NUnit.Framework;
using System.Threading;

namespace DevExpressRepro;

[Apartment(ApartmentState.STA)]
public class Fixture1
{
    [Test]
    public void Test1() => TestContext.WriteLine(Thread.CurrentThread.ManagedThreadId);
}

[Apartment(ApartmentState.STA)]
public class Fixture2
{
    [Test]
    public void Test1() => TestContext.WriteLine(Thread.CurrentThread.ManagedThreadId);
}

(running using ReSharper test runner, NUnit 4.2.2, NUnit3TestAdapter 4.6.0)

[OsirisTerje]

I added your repro code above to https://github.com/nunit/nunit.issues/tree/main/Issue4833
And - I added another fixture.

You are correct with ParallelScope.Children, (or All) everything, both fixtures and tests run on the same thread. If I change to ParallelScope.Fixtures, or Self, everything, both fixtures and tests run in their own separate threads. What is weird, is that even with ParallelScope.None, they run in their own separate threads,

So I wonder if something is broken wrt to Parallel running and the control of the ParallelScope.

If I remove (comment out) the [assembly: Parallelizable(ParallelScope, they are still running in parallel.

I added in the WorkerId, and that stay the same within the same fixture. I believed that each worker ran in its own thread, and not that a worker could run multiple threads.

@CharliePoole You know more about the design of this. Can you comment?

[OsirisTerje]

I did some more accurate testing using the stopwatch .

ParallelScope.Fixtures

The fixtures run in parallel, the tests do not.

ParallelScope.Children

Everything runs sequentially

ParallelScope.Self

Everything runs sequentially

So it doesn't seem like you can run Tests within a Fixture in parallel using STA. I've also tried with different optiosn to each fixture and test, with no change.

[CharliePoole]

@OsirisTerje Keep in mind that I worked on that code four or five years ago. Both the code and I have changed. :-)

However, each worker has a single "worker thread" for running tests but may start a secondary thread for handling timeout. Note that onetime setup and teardown may not necessarily run on the same worker as the test itself.

Unless anything has changed. :-)

[CharliePoole]

@OsirisTerje IIRC @jnm2 did some work in that area and may have further insights.

[CharliePoole]

Somebody else will have to figure out how this works today but I wanted to mention a basic assumption underlying a lot of NUnit historically, which may not be written down anywhere...

At a very early point, back in the first decade of the century, we discussed validating user inputs (including attributes applied) for consistency. We decided explicitly that we would not perform such validation in general but would try to do whatever the user told us to do, even if it resulted in an error. I think it was a reasonable decision back when using TDD and a unit testing framework meant you were working at a pretty advanced level.

Conceivably, now that NUnit is used by developers at every level and even non-developers, that decision may no longer work. A possible intermediate level between complete laissez faire and advance checking for consistency might be to start introducing warnings when the inconsistencies arise.

Anyway, I'm pretty sure there was only ever one single-threaded queue.