Flaky generation of scenarios when using feature files from outside project

[wgala-tomra]

Reqnroll Version

3.1.2

Which test runner are you using?

xUnit 2

Test Runner Version Number

xunit.runner.visualstudio 3.1.5

.NET Implementation

.NET 8.0

Test Execution Method

Visual Studio Test Explorer

Content of reqnroll.json configuration file

{ }

Issue Description

When trying to include feature files that are outside of project, for the first time (and that's not always the case) I see test cases being generated and visible in list of tests.. Following compilations result in feature files not being present in test list. Code behind files are generated but it seems that those aren't included in compilation.

There's one remark from me. If CodeBehind path is similar to what was in outside dir it works quite stable. For example, if there's "$(SolutionDir)Scenarios\FeatureX\featureX.feature" and Project is under '$(SolutionDir)\TestProjectBdd', if that file is placed under '$(SolutionDir)\TestProjectBdd\Scenarios\FeatureX\featureX.feature.cs' then it works quite stable. I found out that because I was using by mistake 'RelativeDir' with 'RecursiveDir' (which seems to be always defaulting to $(ProjectDir)%(RecursiveDir), just side comment)

Steps to Reproduce

• Have default Reqnroll test project in, for example, $(SolutionDir)TestProject\TestProject.csproj
• Have directory that contains feature files in $(SolutionDir)Features***.feature
• Paste following into TestProject.csproj

	<ItemGroup>
		<ReqnrollFeatureFiles Include="..\Features\**\*.feature">
			<CodeBehindFile>ReqnrollCodeBehind\%(RecursiveDir)%(Filename).feature$(DefaultLanguageSourceExtension)</CodeBehindFile>
			<MessagesFile>$(IntermediateOutputPath)ReqnrollMessageFiles\%(RecursiveDir)%(Filename).feature.ndjson</MessagesFile>
 			<Visible>false</Visible>
			<NormalizedLogicalName>$([System.String]::Copy('%(Identity)').Replace('\','/'))</NormalizedLogicalName>
 		</ReqnrollFeatureFiles>
	</ItemGroup>

• Compile first time
  • tests for feature files appear (or may not)
• Compile second time
  • tests are not on the list

Link to Repro Project

No response

[clrudolphi]

At present Reqnroll does not support inclusion of Feature files outside of the project's folder.

This is something that would be nice to support, but is not yet confirmed for the roadmap.

[wgala-tomra]

Just for the sake of what is the use case and what it's needed for, so you know my perspective and may prioritize it better. Maybe you can point me to other direction so I can solve my problem in different way.

For me that feature is a must for implementing BDD in Unit Tests. This is because I have more than one Unit test project going to rely on feature file directory. For now I have two projects, but I plan to expand it to different stacks, so it can be unit tested separately on each stack. Now, the way reqnroll works currently is that it generates code behind (test execution file) at the same place as feature file. I wanted to bypass that or redirect output to project related directory, because if I generate code for TestProject1 then it will be the same as TestProject2.

Again, why wouldn't I, since code will be the same for both test projects anyway? Because I don't want to generate test cases for each scenario in each project. This is related to this #573, where I asked and got help in Tag filtering for scenarios.

For me, not every scenario can be implemented on each stack. Also, tag filtering gives me flexibility and robustness because:

• I can define which tests can be generated.
• Only those tests are running which are marked for run on given TestProject stack. If any step is missing, orthere's typo (basically test is not running), that test will fail. I don't like compromise here, where I find after half year that this test is actually skipped.
• Marked tests should be always running on pipeline, on given stack. Tag will define which stack is covered. Then it can be exported to ADO (which we do) and will give insight for testers whether this is test case that they need to focus or not. I know that UT aren't for testers to take into account but later we may expand on integration tests level, where it's more relevant.
• I can safely have missingOrPendingStepsOutcome set to error, since all covered test cases are running. If tests are not running because of something, they should fail. And that's they role.

I had once an approach to copy scenarios to project folder, so code behind files could be generated there. This had catastrophic flaw, which is that tests in Test Explorer, were pointing to COPY of feature file. So in case someone wanted to add another step, because setup was missing (very frequent case), or just shuffle steps, then all those changes were gone with next compilation. I tried to mark them as read-only, but that didn't give me enough information. I wasn't fine with that, now what will be with someone who stumbles upon that without prior knowledge...

If there's some use case for my problem, or some different way I can tackle this, let me know. Maybe I am missing something and you have thought about such scenario (pun intended) already. Thanks!

[Code-Grump]

There's a strong association between source feature files and the code generated. Reqnroll currently puts the generated source files next to the feature file, in-keeping with how Microsoft's own code-behind generators work.

If we were to do this for two separate projects sharing files, one project's generator would clobber the output of another. If both projects are configured the same, that might not be a problem, but if they're using a slightly different configuration from each other, or even different test runners, the result will be an incorrect source file for all but one project.

I'd like to understand your desire to have the same features in multiple places. It's typical to have a requirement validated by one test suite, where variations are handled within that suite. It's atypical to need to validate a requirement has been met multiple times. If you could elaborate on how you're using Reqnroll, we may be able to suggest other ways to achieve your goals.

TLDR; As of right now, you will need to copy the files between your projects.

[wgala-tomra]

So, first of all. I am not doing Functional/E2E tests. I am trying to apply Reqnroll into Unit Tests.

I can include more details but the important part is that I want to have unit tests on front-end and back-end separately. While back-end is obvious for being able to unit test, we have front-end written in Blazor, which with bUnit library we can execute unit tests (I wouldn't call them pure unit tests though).

We also have MVU architecture where one of the important points we follow is that we can create View from scratch at any time from given Model that is being produced by Update. This is important to understand how BDD relate so much with 'back-end' layer and that front-end derives from model that's being created. And not getting into much details of MVU architecture, we have unit tests for View part (where we test HTML output from given Model and some interactions with user), and unit tests for Update part, where we test how business logic works and reacts to those interactions.

And that's why I was asking about Tag filter generation. While one Scenario can be tested on both layers, other can be tested only on one. And because of that Tag filtering I cannot have common project that will generate tests, and two of them include that one and define step definitions separately. I need to have way to generate two different outputs basing on the test project, but using the same directory of feature files.

[gasparnagy]

I think this request makes sense and I have also seen other folks having similar problems. I think the fist should be to properly support processing feature files outside of the project scope by allowing the generated code-behind files to be saved to the obj folder, instead of the feature file folder. (Initially this should be done in an opt-in way, but in long term this could be the default behavior).

Once I did #861 it was also my explicit goal to prepare for this. Before this PR the target path of the code-behind files were generated by the Reqnroll generator code itself, with that PR this has been modified and now MsBuild calculates the code behind file path (here). The "only" thing we need to do is to calculate this path to the obj folder (similarly to what we do with the MessagesFile metadata) if some MsBuild flag is enabled. The tricky is how you calculate the folder path of the code-behind file when the feature file is above the project root (e.g. when linked as ..\..\SomePath\SomeFeature.feature), but @Code-Grump had some suggestion about that (I just can't remember now) as well.

I am interested to continue on that, but I can only do it in the second half of November. If someone would like to pick this up earlier, that is great.

Once this is done we can continue and see how tag filtering could be better supported, but for that the plugin option that @clrudolphi mentioned is at least a workable solution.

[Code-Grump]

I've long wanted to shift code generation to the obj path, but the last time we tried this experiment, the Visual Studio Test Explorer stopped performing source navigation properly. It might be time to test it again and get on the VS team to fix it if it's still a problem.

[wgala-tomra]

I am quite successful in workarounding this by using hard links. Although, I see some issue with Reqnroll picking it up later (either not authorized exceptiuons or not finding it because of MSBuild glob nature) but I believe what I am facing now is something related more to MSBuild than Reqnroll library.

Now, while I would like to have this built-in, I don't think hard links is something that can be expected from client's machine to be allowed to. On the other hand, from what I tested, if feature files will end up in obj/intermediate folder side by side code behind file, source navigation behaves well. The only thing is that path for source that is opened goes to obj dir, but the content is shared.

[wgala-tomra]

I had time to go back to this issue (it's not my full time focus, but secondary or even tertiary focus).

Hard links work in terms of content. Problem starts when being used with dev environment. Firstly, Git doesn't pick up changes in source file out of the box, which may indicate wrong status of file. Probably need to retrigger file scan. Same with VisualStudio, it doesn't see that content changed and that it should recompile sources. In case of having two projects, while editing in one, second one may not pick up changes, resulting on not building sources on latest content.

This is something I face on Windows, not sure if that's the case on other OSes. The main point is that Windows doesn't trigger folder watcher (or file system event?) that is used by git/VS to listen to the folder changes. Side note: funny scenario is that Notepad++ picks up that change on source file. Even more, if content is refreshed by Notepad++, then VS and git altogether, picks up change notification... No save needed on original file ...

Basically, solution is to use symbolic link instead of hard link, first tests were successful, didn't go deep enough to confirm it though. The reason why I dwell on hard links is that under Windows system theoretically you can make hard links as basic user but symbolic links are domain of admin user. While I understand that symlinks may be security issue due to symlink race, I don't know how that is different in terms of hard links. I am not well versed enough in security to answer that...