nunit-vs-adapter
nunit-vs-adapter copied to clipboard
nunit
Random parameter values cause tests to be not run
Any use of Random, including our RandomAttribute, in the generation of a test case causes the test to be listed as not run in the TestExplorer window. The same tests run as expected in the NUnit runners.
See https://bugs.launchpad.net/nunit-vs-adapter/+bug/1242315 for more info.
I believe this occurs in the adapter because of a fundamental difference in how tests are found when using the adapter as opposed to the NUnit gui.
With the Gui, tests are discovered and then listed in the gui. They are kept loaded and not re-discovered at the time of execution unless you actually call for a reload.
With the adapter, discovery and execution take place in two different processes and no information is shared between them except through the arguments that are passed in. The tests need to be rediscovered at the time they are executed. Usually, this works because exactly the same tests are found. With random parameters, that's not the case. The methods shown as "Not Run" are the random tests created in the initial discovery phase. Other random values are actually used in the execution phase.
To fix this, we would have to change how NUnit presents tests to Visual Studio or VS will have to change it's expectations. So this is being de-prioritized to wait for the next VS release.
Hi CharliePoole, what do you mean when you say "the next VS release" ? Are you speaking about an update to VS2013 or another release ? And, do you think this behaviour will be changed on next VS release or update ?
I meant the following version after 2013. I doubt that ms would substantially change how they deal with tests in the vs2013 context.
It's also possible that we will change how nunit names tests in 3.0, which would influence how the adapter works.
My main point was that this won't be resolved in the context of vs2013 plus nunit 2.x. On Apr 27, 2014 5:09 PM, "Fernando Pérez" [email protected] wrote:
Hi CharliePoole, what do you mean when you say "the next VS release" ? Are you speaking about an update to VS2013 or another release ? And, do you think this behaviour will be changed on next VS release or update ?
— Reply to this email directly or view it on GitHubhttps://github.com/nunit/nunit-vs-adapter/issues/15#issuecomment-41512542 .
Interestingly enough I just updated my WinRT 8.1 VS2013 project to WinUAP 10 VS2015 project. I searched for the error and this is all I found. I never hit this in VS2013. Btw I'm using msunit and not nunit.
I'll added it to V3.0 to mark it for continued effort, but I doubt this can be done within 3.0. I don't think we have anything neither in 3.0 nor in VS 2015 which changes this.
In NUnit 3.0, random parameters are repeatable, provided the same initial seed is used. This is a command-line argument for the console runner. We could do the same thing for the adapter in one of two ways:
- The user gives us a seed to use, via a settings file or some other way.
- We generate a random seed at discovery and save it "somewhere" in order to use the same value on execution. Unless there is some way that VS can accept name/value pairs and give them back to us in the separate process, "somewhere" would probably have to be a file.
I increased the priority from low to normal in the context of 3.0. i agree it's 3.0 only.
I created nunit/nunit3-vs-adapter#97 to track this for the 3.0 adapter. Despite having agreed with it's being 3.0 only last week, I'm not really happy with closing it here. It seems like a pretty fundamental problem, which we ought to fix in the adapter itself. In any case, I dropped it from the (pseudo-)milestone.
Marking this for discussion until it is clear what the future holds for this adapter.
My team is stuck with NUnit 2.x and NUnit 2 Test Adapter. I experienced the issue with a test case source that provided random values. I settled on a solution that called a method to generate a random value from inside the test method and combined this with a repeat attribute. At least I think so, even though it still lists only one test run.
Now I am facing the same problem when using the random attribute. I will try something similar to go around the issue, but it is unfortunate.
Even though I am not able to use it on my current projects, I hope that you are able to address this issue in the NUnit 3 Test Adapter. If possible, also in NUnit 2 Test Adapter.
Yes, the NUnit 3 adapter was addressed in version 3.5. (https://github.com/nunit/nunit3-vs-adapter/issues/97) The fix has evolved a bit over time.
@CharliePoole Is there any complication standing in the way of porting it back to this adapter?
Can we turn this around? What is preventing you @LayZeeDK to upgrade to NUnit 3? Any of the breaking changes? Something else?
Reason is that we don't really expect to release any more updates to 2.X series, if there is not a really compelling reason, like it is breaking in a new VS version.
It seems that we are only able to install and use either NUnit 2 Test Adapter or NUnit 3 Test Adapter in Visual Studio (Professional 2015). As all our previous projects are based on NUnit 2, we have not moved forward with NUnit 3 as it is a hassle to switch between the extensions every time we have to go work on an old/new project.
Am I wrong? I know that recent xUnit versions solve this problem with a specific version of an xUnit runner that is installed as a tooling package dependency for the project.
Rather than installing an extension, sounds like referencing this package will get you what you need: https://www.nuget.org/packages/NUnit3TestAdapter/
Rather than installing an extension, sounds like referencing this package will get you what you need: https://www.nuget.org/packages/NUnit3TestAdapter/ -- @jnm2
Thank you very much, Joseph! I have now run test for NUnit 2 and NUnit 3 projects in the same solution using NUnit3TestAdapter project package and NUnit 2 Test Adapter Visual Studio extension, side by side. I even managed to use a combo of random and value source attributes with the NUnit3TestAdapter package.
Does the NUnitTestAdapter package work with NUnit 2.6.4? Can I run the tests without any Visual Studio extension?
@OsirisTerje can confirm, but I expect that the NUnitTestAdapter works with all NUnit 2.x versions. So you shouldn't need any Visual Studio extension.
Now running NUnit 2.6.4 + NUnitTestAdapter 2.1.1 as well as NUnit 3.9.0 + NUnit3TestAdapter 3.9.0 side by side with multiple projects in the same solution without any Visual Studio test adapter extension. Sweetness!
Now the only task remaining is getting NUnit 3 running on the CI server, which is manageable.
Thank you, gentlemen!
When you're using the nuget adapter packages, they should be picked up on the CI server, if you run vstest.console that is, there is an option to point to the folder where the adapter package is. It is enough to point to the parent folder, it will search all below.
The devs should now uninstall the vsix, since they will override the nuget ones.
And yes, the NUnitTestAdapter (for 2.X runs with all 2.x nunit versions.
The same error occurs in datasource that returns IEnumerable<TestCaseSource> , I use nunit 3
@amado-saladino this repository is for the NUnit 2 adapter. Could you open an issue in the NUnit 3 adapter project and provide more information? IEnumerable data sources are quite common and we haven't seen any issues so please provide an example and information on your configuration (VS version, adapter version, NUnit version, etc).
Thanks
