nunit-console
nunit-console copied to clipboard
nunit
Simplify locating of addins
Currently there is only one way how user is able to say where nunit-console should find extensions. This is via *.addins file. What if user will be able to install extension via nuget into tests project and that's all what is required from user.
I propose to extend nunit-console and try to search extensions in the directory where user's tests dll is located. It would be default behavior.
You can currently do this by either modifying the delivered addins file or (better) adding a new one pointing to that directory.
That may not totally do the job for you but if you try it I think we'll learn better what us needed.
Where is your console coming from at the moment? If you install both console and addins by NuGet, the engine should find the addins automatically already.
Imaging nunit-console came from unknown location, for instance it's pre-installed on build machine (appveyour, etc).
And I do
nunit3-console /full/path/to/my/tests.dll
Actually I don't know where exe is located, hence I cannot use *.addins file.
Generally if you explore some example of the extension https://github.com/reportportal/agent-net-nunit/blob/master/README.md, you might see that the hardest part of the installation is to say nunit-console how to find the extension.
This issue is about the following case:
- User developed tests
- Installed extension via nuget, built/publish project
- Copied all artifacts to some another machine
- Executed tests
It's probably the biggest issue with extensions that there's no easy way to install them manually.
If you use either nuget or (my favorite) chocolatey, they are all findable very easily. But if you copy artifacts, you are basically down to a manual process, which is easy to get wrong.
As you can see from the docs, the process of locating extensions starts wherever the nunit.engine.dll is located. If it is located in your project, because you installed via nuget.org, then any extensions that are also installed by nuget.org are found automatically.
When I use chocolatey to install a copy of the console runner centrally, then the starting point is inside a chocolatey directory. Extensions that are also installed via chocolatey are found, but no others.
When the msi is installed centrally, there is no way to add new extensions except to add new addin files in the the engine directory.
This overall separation of different install methods is actually by design. If you install the engine in your project, it should not be picking up extensions from some central location. However, IMO at least, it would make sense to do as you are asking and have the central installs also look into the project that is being executed for extensions.
Hey @nvborisenko - you've raised a good point. I agree that in your situation there should be an easier way to install extensions. I'm less convinced that searching the test directory for them is the right solution.
As far as I can see, there's a few drawbacks to that approach:
- The engine will now have to examine every assembly in the test directory to look for extensions. In the case of big projects, that's a lot of assemblies, and may be a significant time cost - especially in the case of running a single test.
- It feels like a bit of a semantic oddity to me. Why does something I have as a dependency of my test project affect how the test runner is working?
- What happens if I pass two assemblies into the console runner - with an extension in the test directory of only one of these assemblies? Do all extensions apply to all assemblies? Are the extensions enabled and disabled depending which assembly they were found besides? (Would the latter even make technical sense?)
Would a console command line parameter be a better solution? We already have --list-extensions - how about a --load-extensions? This could take a file path to either a directory, extension assembly, or addins file, to keep it consistent with current extension loading functionality.
Tagging in @nunit/engine-team for their thoughts - let's see what others think about this before going any further. 🙂
It feels like a bit of a semantic oddity to me. Why does something I have as a dependency of my test project affect how the test runner is working?
I've always felt this way about referencing the VSTest adapter from a csproj. :D
--extensions-dir maybe? I like the concept and I agree that we don't want to be scanning the project output being tested.
I've always felt this way about referencing the VSTest adapter from a csproj. :D
Yes - same!
--extensions-dir maybe? I like the concept and I agree that we don't want to be scanning the project output being tested.
And limit it just to take a directory? It seems like an overhead to need to create a separate directory specifically to house a single extension assembly. I think it would be good to also support the .addins file, to be consistent current behaviour.
Oh, true. --load-extensions is better for that. Would it ever be beneficial to disable the default loading search logic? I can't think of a reason.
The search for extensions starts in a root directory, which is the directory containing the engine itself. The idea was that this might be expanded to search in other directories as well, using exactly the same search pattern. One possibility is a common directory used by all engine extensions. Another is some directory specific to the test project being loaded.
However, if you are loading multiple assemblies, there is still only one engine with extensions (i.e. the primary engine) It would take a fair bit of effort, I think, to load an extension only for one assembly, possibly a full restructuring of how we load assemblies. For that reason I agree with an approach where the user simply tells the user where to look for extensions. The command-line makes sense for this purpose. All the console would need to do would be to add the new setting to the TestPackage and the engine should do the rest.
Great! @nvborisenko - does the command line option still work for you, and are you still interested in implementing it?
I suggest --load-extensions can take an extension assembly, a directory, or a .addins file. I think it should be possible to use --load-extensions multiple times, for future-proofing. The load-extensions options will need to go into a new EnginePackageSetting, which can be passed into the engine, and the ExtensionService can then load any relevant extensions.
@ChrisMaddock That flexibility (dll/addins/directory) is probably useful for the user even if it takes a bit more work to reorganize the service initialization code. It seems doable.
You haven't specified what a directory means. I suggest that it should mean the same thing as if you added the directory path to the addins file. That is: process any .addins files if present OR process *.dll in that directory. This is slightly different from how we process the original engine directory, where we ignore any dlls, but it seems less surprising.
Would it ever be beneficial to disable the default loading search logic? I can't think of a reason.
Me neither. Let's cross that bridge when we come it it. 🙂 There's workarounds, like adding an empty. .addins file.
You haven't specified what a directory means. I suggest that it should mean the same thing as if you added the directory path to the addins file. That is: process any .addins files if present OR process *.dll in that directory. This is slightly different from how we process the original engine directory, where we ignore any dlls, but it seems less surprising.
Good point - I agree with what you say!
If we want to run without any extensions I can imagine a --no-extensions option in future. It's possible to enable and disable extensions, but we haven't exposed the possibility in the runner.
Command line option can be considered as working solution. Fifty cents from me:
--load-extensionslooks likebooleanoption, load or not. It's not clear whether applying this option overrides default*.addinsbehavior, will they work together?- Multiple usage: is
--load-extensions path1;path2better than--load-extensions path1 --load-extensions path2?
New idea came: what if engine searches for "*.addins" files in test directory and processes them as usual. Hence user needs create addins file in test project and specify relative path to an extension. This resolve the issue when user doesn't know where exactly nunit-console.exe is located, he knows that an extension will be loaded by runner. This way resolves my initial issue with complicated installation of extension. Moreover, nuget package of extension can add addins file automatically. So user need just install nuget package and that is all, extension will be loaded by engine.
We have discussed but not yet made any decision and nobody has taken the task.
After some time passed, I am returning back. Actually for me it was enough to hear about that "engine starts look up *.addins file in the folder where it is located". It allowed me create addins file and specify where my addin.dll is located. Particular it worked with NUnit3TestAdapter: Visual Studio loads adapter from "bin\Debug" folder, engine.dll is located there too, and my addin.dll is there too.
Using this approach was enough to me. But recently Visual Studio decided to load adapter from nuget package folder, instead of "bin\Debug". As a result approach stopped to work. I lost the actual folder where engine.dll is located. It might be in "packages/NUnit3TestAdapter/*" or in nuget global cache folder. So I don't know where to create addins file.
I understand that extension service should be totally redesigned. What if engine.dll considers Environment.CurrentDirectory to start discovering addins? CurrentDirectory is more predictable directory. In case of NUnit3TestAdapter the directory *.sln directory.
Easy code change - big profit.
The nuget install of NUnit should include a .addins file, which works with extensions installed via nuget. Can you do something with that?
It seems to me that CurrentDirectory works for the adapter, at least until VS changes the directory, but doesn't work well generally. For example I can run a command like...
..\..\nunit3-console some\directory\mytest.dll
OTOH I've often wanted a way to have NUnit detect that it's running as part of a VS project and use additional addins locations in that case.
NUnit3Adaper comes with addins file, by default it includes all "near" nuget packages.
Problems:
- It's unclear what is version of nuget package is picked up by engine
- Addin package has to ship all dependencies inside to be loaded correctly
Let me remember the topic of this issue: simplify locating of addins. I want to develop nunit extension (which is delivered as nuget package because of this is the easiest way to deliver something in .net world). User is free to install this nuget package into tests project, and nunit engine should use this package as addin. Any special requirements for package?..
To be more specific: I want to develop "html reporter". User has a test project, user installs "html test reporter" into his test project via nuget - and he gets html report in all ways he executes tests (via console, via adapter). Is it possible?
I agree that a simplified approach is a good idea. I just don't think the Current Directory is reliable in all cases since the user can be in any directory and still run NUnit3-console.
At least for the console, it seems to me that the application base directory would be more reliable. Not sure how that would work for the adapter, however.
Ooh, glad to see people interested in this issue - it's something I've wanted to look at for a while!
My thoughts: the engine itself provides a simple interface to locate extension. How to make that more user-friendly is a situation which is different per runner - any my opinion is that any changes we want to make here would be to the runners, (i.e. the adapter, or console).
For example:
- Based on your comments, Nikolay, the adapter may want a different default path, set to wherever NuGet packages installed into a particular project end up. This would be achieved through pacaking (or dynamically modifying) the addins file. (Or perhaps via direct integration with the ExtensionService.) Either way - this would possibly rely on information only available in the adapter.
- For the NUnit Console, I've long since dreamed of an
nunit3-console install-addin path/to/myaddin.dlltype of command - A GUI runner such as ReSharper/TestCentric may just wish to instead just pop up a file explorer window, let the user select the relevant assembly, and append it to the addin's file.
For this particular GitHub issue - I'd love to see someone take on a console command such as the above, but I don't (yet!) think any change to the engine is necessary. On the adapter - I'd defer to @OsirisTerje, and suggest a separate issue over in that repo to look at the adapter implementation specifically. 🙂
All that said, the point of @nvborisenko's I haven't touched on is being able to install a dependency and have that add-in apply to runs extensions of that test.
I'm not sure on this - it's sounds potentially costly performance wise. Remember the engine and extensions are all invoked whenever you run a single test in the adapter/ReSharper. It sounds like it could be a little heavy to investigate every reference of the test assembly for possible NUnitExtensionAttribute's before running a single test every time...but I haven't tried it in practice...
Adding a small point to the mix here...
My expectation was that folks would not modify the provided addins file but would add additional files, perhaps one file per extension, in the same directory. That's why we initially start the search for extensions with a directory rather than a single file.
A couple of ideas for introduction of new extensions more easily...
-
Provide a new method for adding new directories to the initial one that is searched. This would need to be added to the extension service interface. I already did something similar for adding assemblies to be searched for extension points. (not sure if it's in the NUnit engine, however)
-
Specify either "NUnitExtensionDir" or "NUnitExtensionPath" as an environment variable. "Dir" would be one directory. "Path" could be multiple semicolon-separated directories. Installing a new extension would simply involve dropping a
.addinsfile into a directory. This seems ultra simple to implement and wouldn't change any interfaces. However, if the engine is checking this variable, it will mean that all runners get the same extensions. I don't know if that's a feature or a bug!
Here's number 3, which is merely a possible workaround...
- Find the primary
.addinsfile for the console on your machine. Add another.addinsfile in the same place with a single line that points to a directory where you will drop the.addinsfiles for additional extensions. The engine will find those extensions. What I don't know is how this will work across nuget restores or on your CI machines.
My expectations:
- Extensions should NOT be machine scoped. So we don't want to use environment variables, or any other global settings. We don't want to affect whole development environment.
- Extensions should NOT be runner specific. We should have ability to use any set of extensions, when executing
TestsA.dll; and any other predefined set of extensions, when executingTestsB.dll.
Based on this, I would like to say that runner/engine should be smart enough to understand what extensions ProjectA or ProjectB want to use.
I think the first point is overstated. Environment variables may be machine-scoped but may also be user-specific. It depends where they are set and by whom.
The second point is true when the extension point for which the extension is written is part of the engine. But if the extension point is in a runner, then extensions have to be runner-specific. Note that right now, extensions are specific to a particular copy of the engine. So their specificity depends on how engines are located and used. For example, extensions to the copy of the engine that is installed by the VS adapter are runner-specific and (usually) user-specific.
@nvborisenko I'm not arguing for or against a particular approach here. I'm only trying to point out that there are lots of different sides of the problem and that it's probably not a good idea to make absolute rules saying that anything MUST or MUST NOT be done a certain way. We can each only say what type of setup would be helpful to us.
Moving this to the 4.0 milestone for resolution. While it could probably be done in a non-breaking way, I think being allowed to break existing behavior in a new major release gives us a bit more freedom to rethink the loading of extensions.