nunit3-vs-adapter
nunit3-vs-adapter copied to clipboard
nunit
Operation could destabilize the runtime
- NUnit Adapter 3.10.0.21
- TFS 16.131.27701.1
- Code coverage enabled = true
NUnit Adapter 3.10.0.21: Test execution started
2018-06-18T12:18:41.2125333Z Running all tests in E:\Agent\_work\209\s\src\*****.Tests.Unit.dll
2018-06-18T12:18:42.8180637Z Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in E:\Agent\_work\209\s\src\**************.Tests.Unit.dll
2018-06-18T12:18:42.8180637Z The NUnit 3 driver encountered an error while executing reflected code.
2018-06-18T12:18:42.8620107Z at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
2018-06-18T12:18:42.8620107Z at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
2018-06-18T12:18:42.8620107Z at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
2018-06-18T12:18:42.8629873Z at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
2018-06-18T12:18:42.8629873Z at NUnit.Engine.Runners.TestDomainRunner.LoadPackage()
2018-06-18T12:18:42.8629873Z at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
2018-06-18T12:18:42.8629873Z at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter)
2018-06-18T12:18:42.8629873Z at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)
2018-06-18T12:18:42.8629873Z at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, TestFilter filter) in D:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 229
2018-06-18T12:18:42.8629873Z Innerexception: System.Security.VerificationException: Operation could destabilize the runtime.
2018-06-18T12:18:42.8629873Z at NUnit.Framework.Api.FrameworkController..ctor(String assemblyNameOrPath, String idPrefix, IDictionary settings) in C:\src\nunit\nunit\src\NUnitFramework\framework\Api\FrameworkController.cs:line 68
Can you please upload a repro project? Make it as small as you can, it should compile "as is".
Perhaps try reducing the internal project, e.g. remove code so long as it still reproduces. If you have a minimal reproducer with your production code, the next step would be to "rewrite" it so you can publish it here. Otherwise I'd guess it is really hard to analyse the issue without further information.
We need some more information on this, the normal way is that you provide a similar working repro project. We dont need to see anything from your internal project, but we need to know what you are trying to do. Running a normal test project with VSTS (which is ahead of TFS dev 16) works as it should, using the ordinary VSTest task.
(Thanks @siprbaum , you beat me to it :-) )
I reproduce it use empty nunit project
packages.config
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="NUnit" version="3.10.1" targetFramework="net472" />
<package id="NUnit3TestAdapter" version="3.10.0" targetFramework="net472" />
</packages>
UnitTest1.cs
using System;
using NUnit.Framework;
namespace UnitTestProject1
{
[TestFixture]
public class UnitTest1
{
[Test]
public void TestMethod1()
{
Console.WriteLine("test");
}
}
}
b7b3e470-73cd-11e8-ba0f-2dd83a1f0155.runsettings
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<ModulePaths>
<Include>
<ModulePath>.*.dll$</ModulePath>
<ModulePath>.*.exe$</ModulePath>
</Include>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
</Exclude>
</ModulePaths>
<Functions>
<Exclude>
<Function>^Fabrikam.UnitTest..*</Function>
<Function>^std::.*</Function>
<Function>^ATL::.*</Function>
<Function>.*::__GetTestMethodInfo.*</Function>
<Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
<Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
</Exclude>
</Functions>
<Attributes>
<Exclude>
<Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>
<Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>
<Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>
<Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>
<Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
</Exclude>
</Attributes>
<Sources>
<Exclude>
<Source>.*\atlmfc\.*</Source>
<Source>.*\vctools\.*</Source>
<Source>.*\public\sdk\.*</Source>
<Source>.*\microsoft sdks\.*</Source>
<Source>.*\vc\include\.*</Source>
</Exclude>
</Sources>
<CompanyNames>
<Exclude>
<CompanyName>.*microsoft.*</CompanyName>
</Exclude>
</CompanyNames>
<PublicKeyTokens>
<Exclude>
<PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
<PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
<PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
<PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
<PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
<PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
<PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
</Exclude>
</PublicKeyTokens>
<UseVerifiableInstrumentation>False</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>False</CollectAspDotNet>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
run.bat
e:\15.8.0-preview-20180610-02\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "UnitTestProject1.dll" /logger:"trx" /Settings:"b7b3e470-73cd-11e8-ba0f-2dd83a1f0155.runsettings" /TestAdapterPath:"e:\Test\Debug\"
Microsoft (R) Test Execution Command Line Tool Version 15.8.0-preview-20180610-02
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
NUnit Adapter 3.10.0.21: Test execution started
Running all tests in e:\Test\Debug\UnitTestProject1.dll
Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in e:\Test\Debug\UnitTestProject1.dll
The NUnit 3 driver encountered an error while executing reflected code.
at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
at NUnit.Engine.Runners.TestDomainRunner.LoadPackage()
at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter)
at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)
at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, TestFilter filter) in D:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 229
Innerexception: System.Security.VerificationException: Operation could destabilize the runtime.
at NUnit.Framework.Api.FrameworkController..ctor(String assemblyNameOrPath, String idPrefix, IDictionary settings) in C:\src\nunit\nunit\src\NUnitFramework\framework\Api\FrameworkController.cs:line 68
NUnit Adapter 3.10.0.21: Test execution complete
No test is available in e:\Test\Debug\UnitTestProject1.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
- Can you zip together the project and upload that, the code above should work.
- Does it also fail when you don't use the runsettings file?
- Does it also fail when you change the .net framework target to e.g. 4.7.1 or 4.6.1 ?
- Does it also fail within Visual Studio ?
Ok, I can confirm it now. It is caused by the runsettings file.
If you change your runsettings file to use InProcDataCollectors instead it will work.
This needs to be properly documented. I'll add a seperate issue for that.
file b7b3e470-73cd-11e8-ba0f-2dd83a1f0155.runsettings is generated by VSTest task I can not change it manual
Yes, you can change it:
(MS also has noted the current behaviour as a bug, see links from @pvlakshm above. )
@OsirisTerje
The UseVerifiableInstrumentation is set to false by the VsTest task when the platform is acquired via the vstestplatform tool installer (gets the nuget package) as the profiler may not be available in the GAC and needs to be resolved using certain environment variables set by the VsTest task.
This is a must for code coverage to work when there is no VS installed on the target machine.
This will have to be fixed otherwise code coverage will not work for NUnit tests when run using the VsTest Platform Nuget package (No VS).
Thanks @ShreyasRmsft
With "have to be fixed", do you mean in the VSTest task or will the project need its own runsettings file?
I added custom runsettings with <UseVerifiableInstrumentation> True </ UseVerifiableInstrumentation> Nothing changes
@OsirisTerje is the NUnit adapter throwing this exception? Is the adapter reading the data collector node in the runsettings?
@ShreyasRmsft It is an exception coming from the NUnit Engine, so it is not the adapter itself, but the embedded NUnit engine. I am unsure about how this collector is intended to work, one of you guys did these changes earlier, and it seems that something here causes the engine to crash. However, when I changed it to InProcDatacollector my repro worked.
And yes, the adapter reads the node and then sets some other properties:
@NikitaEgorov Did you change the runsettings as I described above? (not only setting the flag to true, but replacing the DataCollector with InProcDataCollector?
@OsirisTerje changing it to inproc is not the solution. Ideally in this case the engine should not be reading these settings at all as they are specific to code coverage.
@ShreyasRmsft The engine don't read MS properties, only NUnit properties. The adapter does the conversion as shown above. Anything else being done must happen on the datacollector side. And this code above is from MS, so I assume you are setting whatever is needed correctly here ?
The exception surfacing in the Engine doesn't seem to have anything to do with the Engine itself, but with whatever it is trying to load:
@OsirisTerje I do not have a lot of context on the change that was done.
But yes if the change that was made by one of us is trying to read this settings then it needs to be fixed. I haven't had time yet to see where exactly in the code this is happening. But Nunit should ideally ignore anything present in the code coverage node. Will get back to you on the fix.
@ShreyasRmsft I believe the code involved were written by @navin22, i think this PR #391 was the essential one. I also see a later modification by @drognanar to "Use InProcDataCollector setting to enable sequential run", PR #510, but this is not released yet, it's for 3.11.
Hi there,
I have been told that this issue affects another issue (which I submitted). I am quite new to this repo, so I cannot get what is the current status. Is anyone working on a solution to this issue?
Thanks!
@MoiMoiDG MS themselves are working on it on their side, but I'll check up on it.
@ShreyasRmsft Can you mail me on what I need to do with the adapter to have this fixed? (terje at hermit.no)
@ShreyasRmsft
Since we use sourcelink ( and many more do the same), using the pdb’s as a indicator for the “dll’s under test” is likely to cause more such issues. And the test framework dll's should not in any case be part of the profiling.
Do you call the profiler before or after you call the adapter? Can we add the nunit dll’s to the exclude filter in the adapter ?
Sorry for such delay on this.
Issue Explanation: When we instrument a dll which contains methods/classes marked with certain Security Attributes e.g. SecurityCritical or few more, then during code coverage these methods are likely to throw exception, since these attributes prohibit certain scenarios, e.g run-time modification of existing code, or writing to raw memory, etc.
Solutions: We do have fix for such scenarios, where we delegate responsibility of writing to raw memory, to another dll, which we call Shim dll, which gets installed in GAC when VS is present, however in XCopyable scenario this fails.
We are planning to include Shim dll as part of our XCopy package, & also we plan to release code coverage nugets, which will allow users to take explicit dependency on Shim dll. I'll update the thread once we have done either.
For now only solution available to user would be use code coverage runsettings which will exclude NUnit dlls from getting instrumented.
Hi @mayankbansal018,
Thanks for explaining the problem!
For now only solution available to user would be use code coverage runsettings which will exclude NUnit dlls from getting instrumented.
Could you please provide an example of such runsettings file where it works? I know it is a workaround, but it would be helpful.
Thanks!
@MoiMoiDG the following runsettings should help you
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <RunSettings> <DataCollectionRunSettings> <DataCollectors> <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"> <Configuration> <CodeCoverage> <ModulePaths> <Include> <ModulePath>.*.dll$</ModulePath> <ModulePath>.*.exe$</ModulePath> </Include> <Exclude> <ModulePath>.*CPPUnitTestFramework.*</ModulePath> <ModulePath>.*nunit.*</ModulePath> </Exclude> </ModulePaths> <Functions> <Exclude> <Function>^Fabrikam.UnitTest..*</Function> <Function>^std::.*</Function> <Function>^ATL::.*</Function> <Function>.*::__GetTestMethodInfo.*</Function> <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function> <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function> </Exclude> </Functions> <Attributes> <Exclude> <Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute> <Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute> <Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute> <Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute> <Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute> </Exclude> </Attributes> <Sources> <Exclude> <Source>.*\atlmfc\.*</Source> <Source>.*\vctools\.*</Source> <Source>.*\public\sdk\.*</Source> <Source>.*\microsoft sdks\.*</Source> <Source>.*\vc\include\.*</Source> </Exclude> </Sources> <CompanyNames> <Exclude> <CompanyName>.*microsoft.*</CompanyName> </Exclude> </CompanyNames> <PublicKeyTokens> <Exclude> <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken> <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken> <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken> <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken> <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken> <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken> <PublicKeyToken>^E361AF139669C375$</PublicKeyToken> </Exclude> </PublicKeyTokens> <UseVerifiableInstrumentation>False</UseVerifiableInstrumentation> <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses> <CollectFromChildProcesses>True</CollectFromChildProcesses> <CollectAspDotNet>False</CollectAspDotNet> </CodeCoverage> </Configuration> </DataCollector> </DataCollectors> </DataCollectionRunSettings> </RunSettings>
Hi @mayankbansal018,
Thank you!
However, I still get the error in attachment. I attached the full solution too, for your convenience (it is just a small sample) ... :( 7_VsTest - testAssemblies.zip ESXD-NUTests.zip