SpecFlow
SpecFlow copied to clipboard
SpecFlowOSS
Move generated files outside of project source
Removes the generated codebehind files from the project structure. Generated files are now written to a subdirectory in the /obj/ folder.
- Change MSBuild targets for build and clean to generate files as a build artefact, prior to compilation.
There will need to be some advice given on upgrading, i.e. all feature.cs files should be removed from the project folders, and the documentation at https://specflow.org/documentation/Generate-Tests-from-MsBuild/ could possibly benefit from some information about the few properties available to customize the process.
Types of changes
- [ ] Bug fix (non-breaking change which fixes an issue).
- [x] New feature (non-breaking change which adds functionality).
- [x] Breaking change (fix or feature that would cause existing functionality to not work as expected).
- [ ] Performance improvement
- [ ] Refactoring (so no functional change)
- [ ] Other (docs, build config, etc)
Checklist:
- [ ] I've added tests for my code. (most of the time mandatory)
- [x] I have added an entry to the changelog. (mandatory)
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
It looks like the build failure might be transient. Is there any way to run the failed test again?
Since another fix, we get this kind of error sometimes in our CI pipeline. No idea yet why. You can ignore it.
I will try to find some time on Monday/Tuesday to review the PR.
I tried out the latest version of this PR.
Sadly as I assumed, moving the files into the obj folder breaks the navigation from the Test Explorer to the feature file. This is working for SDK- style projects, but after this change, this is not working anymore.
I have here a solution where I tested it. It has projects for every unit test framework in old and SDK- style project format.
As far as I know and remember the test explorer has some code for SpecFlow to make this navigation work. What I don't know if this depends on the DependentUpon property of the Compile item or simple that the file is in the same folder.
When I tested it for the old project format, I also saw, that the code-behind file is not generated in the obj folder.
Thanks for checking it out, @SabotageAndi.
The failure to generate feature-files for classic project formats is pretty unexpected. I'm going to guess that the property I've tried to use wasn't defined in the MSBuild version you used; it should be an easy fix using different MSBuild props. I will focus on getting this working first.
When I tested locally, the source-link was working for SDK-style projects; I did not test for classic projects. I will have to try repeating my tests locally. I thought this was driven primarily by having a #line directive in the generated feature files. I suspect this may be the harder of the two issues to resolve.
I've cloned and built the master branch of the test project as a baseline. In the cases where source link is working, I can see #line directives being emitted in the code-behind files.
Looking at the files being generated by my branch, there are no #line directives. Having looked at some other projects using the current release of SpecFlow, I can see they also don't have these #line directives and display "No source available" when examined in the test explorer.
Is there something specifically required to have these directives emitted? If I can see it in production use, I'm wondering if I've unexpectedly tripped this behaviour.
Is there anything external contributors can do to assist with this issue? It's quite annoying for most of the developers in our team and it would be nice to fix it.
The outstanding issue is having the Visual Studio Test Explorer be able to link the test to its source source (the .scenario file). If anybody could produce a source file in the /obj path which worked to link back to a scenario file, the generator could be updated to produce it and the ticket could progress.
The issue here I think is because the files are generated at build time and Test Explorer does not read the pdb. This can be confirmed by opening the generated files in visual studio, you will notice that the files are not part of any project. The code generation needs to happen at design time for test explorer to recognize them. This also means that the files will be generated whenever the feature file is saved.
The issue here I think is because the files are generated at build time and Test Explorer does not read the pdb.
The current code-generation also runs at build-time, but generates the files within the scope of the source files. What makes you believe the PDB is the issue? Could you demonstrate this?
I am not saying PDB is the issue. When files are generated at build time, Visual Studio does not know they are part of project. This can be proved by opening the generated files in Visual Studio and the project dropdown on the top left of the editor window showing 'Miscellaneous Files'. Since the generated files are not part of project, Test Explorer is also not able to find the source of the test.
I tried this in a different way and it partially works. Currently the code generation targets run before the Build target. If I change that to run them before BeforeCompile target, Visual Studio (and Test Explorer) correctly recognizes the generated files as being part of project. However, navigation to the source still fails and it looks to be an issue with Test Explorer itself. I can show you what my changes look like if you are interested.
At present, everything works correctly with this feature branch except the navigation to source in Test Explorer. The expectation is that the explorer should link to the feature file, not the generated files, but this is proving to be nearly impossible to diagnose without having a deeper understanding of the test explorer itself.
A shameless nerd-snipe, but I wonder if @nohwnd can help us with understanding how the navigation with Test Explorer works or can point us in the direction of someone who can.
Is there a simple repro we can use to investigate this?
@shyamnamboodiripad Is there a way to redirect the test object to a different source than the one that is used to run the test? I doubt there is. If there isn't should this be made part of the FQN effort?
There's no simple repro for this at the moment, but I could create a repo which simulates the generator:
- Having a feature file in the source tree
- Including an "external" source file in the compile process which uses #line directives to map to the feature file
The current SpecFlow generation process (which includes the generated files within the source tree) does link correctly via the Test Explorer. We'd like to be able to hook into that same process.
At present, everything works correctly with this feature branch except the navigation to source in Test Explorer. The expectation is that the explorer should link to the feature file, not the generated files, but this is proving to be nearly impossible to diagnose without having a deeper understanding of the test explorer itself.
When I tried with the changes in this PR, Test Explorer did not even show the feature file name that the test was generated from. I got it to show the feature file name and line number.
I got this working. Initially I tried my changes in master branch. When I applied my changes on top of this PR, navigation from Test Explorer to feature file worked exactly as expected.
I got this working. Initially I tried my changes in master branch. When I applied my changes on top of this PR, navigation from Test Explorer to feature file worked exactly as expected.
Can you identify which changes specifically made a difference or provide a working branch?
Core of my changes are around when the generation targets run. I forked from your repo. I will commit the changes and let you know.
I created a PR in your fork. Either take that or take changes manually, its up to you. But I only verified with my usage. You may want to verify it more.
Thanks, @nvmkpk I'll try incorporating these changes and see how the unit tests hold up.
@SabotageAndi do you have any feelings towards moving the code-generation from "before build" to "before compile"? It's slightly later in the build sequence, but it may solve the Test Explorer problem. We should still be able to insert the BeforeGenerateSpecFlowCodeBehindFiles and AfterGenerateSpecFlowCodeBehindFiles targets around the generation target.
@Tragedian I am ok with moving it to "before compile". There is nothing that I can remember that would say something against it.
Running the modifications locally, I can see Test Explorer now links successfully to the generated source files:
@nvmkpk is this what you also see when running the integration test-suite?
@Tragedian @nvmkpk let me know when you are finished and I can review and try out the PR.
I'm seeing a large volume of test-failures locally. I'll work through them as best I can, but it's possible I just have a mis-configuration in my environment. @SabotageAndi is the SpecFlow repo no longer running automated tests on PRs? I treated them as my canonical test results previously.
@Tragedian For PRs the automated tests are still running. Perhaps GitHub isn't triggering it, because there is a merge conflict.
@nvmkpk is this what you also see when running the integration test-suite?
Yes but I tried it with my own test project. I was able to navigate to feature file by double clicking on the test in left tree and also by clicking on the source file name link in the right pane.
