Run tests last - Order attribute case

[MPasadu]

While having the order attribute is great for tests that need to run first / or in a given order, it would be great to also have an attribute for running specific tests last.

Our solution looks like this:

[Test, Order(1)]
public void TestA() { ... }

[Test, Order(2)]
public void TestB() { ... }

[Test]
public void TestC() { ... }

[Test, {shouldRunLast in order(1)}]
public void TestD()

We have some quick integration tests which should run first. We have "normal" tests which don't have the order attribute and we have some very long running tests that should run last.

You can see that it would be a problem to give all "normal" tests an arbitrary order number just so the long running tests are ran last. A "LastOrder(n)" attribute would be appreciated.

[CharliePoole]

A separate attribute means we would have to resolve conflicts if both appeared. What about re-defining to negative order values mean relative to the end. That is, -1 would mean run last, -2 would run right before those, etc. Technically, it's a breaking change, but I can't see why anyone would be setting negative values as the attribute is now documented.

[MPasadu]

That would be a possible solution. So:

order(1) -> order(2) -> [noOrder] -> order(-2) -> order(-1)

Edit: What would happen with order(0) though?

[rprouse]

I like @CharliePoole's idea of negative order which mirrors array access for some languages. I would assume that Order(0) would continue to be treated as the first item run.

I am curious about the current behavior of unordered tests with ordered tests. I expect they run after ordered tests, but I'm not sure we ever specified that. If we go with @MPasadu's suggestion, then what about if they are also parallel? They will be run on the parallel queue and the negative tests running on the single threaded queue will need to pause and wait before they start up again. Knowing that code, i'm not sure I like the complexity it would introduce.

[mikkelbu]

I am curious about the current behavior of unordered tests with ordered tests. I expect they run after ordered tests, but I'm not sure we ever specified that.

On https://github.com/nunit/docs/wiki/Order-Attribute we state "Tests with an OrderAttribute argument are started before any tests without the attribute.". Also this corresponds with the code in src\NUnitFramework\framework\Internal\Execution\WorkItemBuilder.cs where we insert items with order in the front of the list of workitems and afterwards sort this subsequence using WorkItemOrderComparer. How this relates to parallel I don't know.

[CharliePoole]

Probably not formally specified but previous discussions assumed that ordered tests ran after unordered. It's implemented by use of MAXINT as the default order value.

The logical extension would be to use a custom comparison so that negative order is treated as greater than positive.

Order has no knowledge of parallelism and simply prioritizes the start of tests. I think we should keep it that way and devote energy to a true implementation of test dependencies rather than making further minor improvements to order. Order can be used to simulate dependency in the following cases...

1. Non Parallel tests
2. Tests in STA
3. Single-threaded fixtures