"Assembly not supported" for .NET 3.5 test project when extension directory contains .NET 4+ assembly

[CharliePoole]

We are having problem with the test adapter in our TFS build environment. Here are the versions used:

• NUnit version 2.6.2
• NUnit test adapter version 0.96.0.4
• Visual Studio 2012 update 3
• TFS 2012 update 1

Here are the steps to reproduce:

• Create a .NET 3.5 class library project with at lease one NUNit test. In this example it is called "aaa.bbb.Tests.dll"
• Configure your build environment according to this guide: http://www.mytechfinds.com/articles/software-testing/6-test-automation/72-running-nunit-tests-from-team-foundation-server-2012-continuous-integration-build

Basically you copy the test adapter assemblies to a folder reachable by the build controller and edit the "test assembly file specification" in the build definition to match the name of your NUNit test project.

• Queue a build, and wait for it to finish.

Result: You should see correct test results in the build report.

However, the folder specified in "version control path to custom assemblies", where the test adapter assemblies are copied, is used for more than just test adapters. For example you place assemblies that contains workflow activities for customized build workflows. And here comes the strange part:

• Create a new .NET 4.0 class library project in visual studio. Name it something that comes before "nunit" in the alphabet, for example "abc.dll". In our case it contains workflow activities, but we can reproduce the problem with just the the standard "Class1.cs".
• Build the .NET 4.0 project and check in the resulting "abc.dll" to the "version control path to custom assemblies" folder. That is the same folder as the NUNit test adapter assemblies were copied earlier.
• Queue another build.

Result: No tests are run and the build report says:

Assembly not supported: C:\Builds\5\aaa\bbb\bin\aaa.bbb.Tests.dll No test is available in C:\Builds\5\aaa\bbb\bin\aaa.bbb.Tests.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

We have found the following ways to make it work again:

• Remove the "abc.dll" from the custom assemblies folder.
• Rename "abc.dll" to "xyz.dll" (!)
• Change the target framework version of "abc.dll" to .NET 3.5.

We can also reproduce the problem locally in Visual Studio, using the same project setup as above. This is a strange thing to do, but much easier for troubleshooting.

• Install the latest NUnit test adapter for Visual Studio.
• Open the .NET 3.5 class library project containing NUnit tests.
• Open the Test Explorer window and build the project. Your tests should appear.
• Right click some of the tests and select "Run selected tests".

Result: Your tests should run successfully.

Now look up the Visual Studio extensions folder, where the test adapter is installed. On my computer it is C:\Users\username\AppData\Local\Microsoft\VisualStudio\11.0\Extensions\do0tbn4e.opv.

• Copy the "abc.dll" to the test adapter folder you found above.
• Restart Visual Studio, open and build the NUnit project

Result: The tests are discovered in the Test Explorer window as before, but when you right click a test and select "Run selected tests" you get the following error message in Output/Tests:

NUnit 0.96.0.4 executing tests is started Assembly not supported: c:\Projects\aaa\bbb\bin\Debug\aaa.bbb.Tests.dll NUnit 0.96.0.4 executing tests is finished

We find it very strange that an unreferenced .NET 4.0 assembly in the same folder as the test adapter can disturb the test execution. And we have really no choice to alter our .NET versions. The NUnit test projects need to be 3.5 for legacy reasons and the workflow activity assemblies needs to be .NET 4.0, or actually .NET 4.5, to reference the TFS 2012 assemblies.

More discussion at https://bugs.launchpad.net/nunit-vs-adapter/+bug/1209192

[CharliePoole]

@OsirisTerje Seems like we would fix this if confirmed. Do you disagree?

[OsirisTerje]

Agree