xUnit 3 Non-parallelizable collection attribute not added

[ChristiaanNel]

Reqnroll Version

3.2.1

Which test runner are you using?

xUnit 3

Test Runner Version Number

3.2.0

.NET Implementation

.NET 8.0

Test Execution Method

Visual Studio Test Explorer

Content of reqnroll.json configuration file

{
  "$schema": "https://specflow.org/specflow-config.json",
  "generator": {
    "addNonParallelizableMarkerForTags": ["non_parallel"]
  }
}

Issue Description

When using Reqnroll.xunit.v3, the [Collection("ReqnrollNonParallelizableFeatures")] attribute isn't added to the generated feature classes.

Could it be because the XUnit3TestGeneratorProvider does not include UnitTestGeneratorTraits.ParallelExecution like XUnit2TestGeneratorProvider does? Currently I can bypass this by creating a partial class for the feature and add the attribute there manually.

For comparison:

xUnit3

https://github.com/reqnroll/Reqnroll/blob/main/Plugins/Reqnroll.xUnit3.Generator.ReqnrollPlugin/XUnit3TestGeneratorProvider.cs#L39

xUnit2

https://github.com/reqnroll/Reqnroll/blob/aeb2cc46b05f794a44643b6faed3601b42ac956e/Reqnroll.Generator/UnitTestProvider/XUnit2TestGeneratorProvider.cs#L50

Steps to Reproduce

1. Create an xUnit test project using Reqnroll.xunit.v3
2. Add a tag to addNonParallelizableMarkerForTags
3. Add the same tag to your feature
4. Inspect the generated feature class

Link to Repro Project

No response

[clrudolphi]

@ChristiaanNel 's analysis does look correct.

After digging around a bit, it seems that we neglected to include an xunit v3 version of the tests that exist for xunit2 (see this class). If we had, I believe this omission would have been found.

We also appear to have missed this for the MsTestv4 support.

There is a bit of a complication though, and I'd like @reqnroll/core-team input before proceeding. The Reqnroll.GeneratorTests project is an xunitv2 test project. Normally, that project takes a dependency on Reqnroll.Generator, which has the implementations of the IUnitTestGeneratorProviders. But starting with xunitv3 we began locating the IUnitTestGeneratorProviders implementations directly in their respective generator plugins projects. In this case, we have the XUnit3TestGeneratorProvider class in the Reqnroll.xUnit3.Generator.ReqnrollPlugin project. That project has a direct dependency on the xunitv3 assembly.

As a result, we can't test the XUnit3TestGeneratorProvider using the Reqnroll.GeneratorTests project because that will cause a namespace collision (for instance, the [Fact] attribute is from the Xunit namespace and is present in both xunitv2 and xunitv3 assemblies). That is, we can't test xunitv3 code using xunitv2 as that would require both assemblies loaded at the same time.

I can think of two paths forward and would like your input:

1. Use reflection to load the Reqnroll.xUnit3.Generator.ReqnrollPlugin assembly, rather than a fixed dependency. This would require a known relative path between the projects. I'm not confident on this being long-term feasible.
2. Move the XUnit3TestGeneratorProvider class to the Reqnroll.Generator project, alongside the other existing providers (i.e., give up on the idea of locating the generator providers within the generator plugins themselves).

I'm open to other ideas.

[Code-Grump]

If the problem is test-related, we should change the structure of our tests. I'd prefer to see a one-to-one relationship between test-framework adapters and a test suite to verify the adapter, instead of a single, central project trying to "cross the streams" as it were.

[clrudolphi]

If the problem is test-related, we should change the structure of our tests. I'd prefer to see a one-to-one relationship between test-framework adapters and a test suite to verify the adapter, instead of a single, central project trying to "cross the streams" as it were.

Fair enough; refactoring in that direction is a good idea. Doing this would involve moving the various IUnitTestGeneratorProvider implementations out of Reqnroll.Generator and in to their respective generator plugin project. Then we would add a test project per test framework generator plugin, and move the respective tests out of Reqnroll.GeneratorTests.

For now, and to speed along a solution to this current issue, I'll take that approach for xUnit3 and leave the refactoring of the test project structure for other test frameworks for later.

[gasparnagy]

Released with v3.3.0 today: https://github.com/reqnroll/Reqnroll/releases/tag/v3.3.0