Multiple frameworks doesn't work with VS test explorer

[h0useRus]

Hi, latest version working fine with one framework with Test Explorer and dotnet test, but if you select more than one framework in project.json working only dotnet test, but not in Test Explorer, all tests disappear. My project.json:

{
    "version": "1.2.3.4",
    "dependencies": {
        "MyLib": {
            "target": "project"
        },
        "NUnit": "3.2.1",
        "dotnet-test-nunit": "3.4.0-alpha-2"
    },
    "testRunner": "nunit",
    "frameworks": {
        "net451": { },       
        "netcoreapp1.0": {
            "imports": [
                "dnxcore50",
                "netcoreapp1.0",
                "portable-net45+win8"
            ],
            "dependencies": {
                "Microsoft.NETCore.App": {
                    "version": "1.0.0-*",
                    "type": "platform"
                }
            },
            "buildOptions": {
                "define": [ "NETCORE" ]
            }
        }
    }
}

[rprouse]

Thanks, I will test this and see if I can figure out what is going on.

[rprouse]

@h0useRus I had feedback today that this is a known issue with all test runners in Visual Studio with .NET Core at the moment. I am going to mark this as blocked since it is a Visual Studio issue, not an NUnit issue as far as I know.

[rprouse]

@h0useRus I have updated a test repository at https://github.com/rprouse/NUnitWithDotNetCoreRC2 with multiple targets. I am seeing tests in test explorer, but I am only seeing one set of tests and I don't know what framework they target. As I said, that is a known issue.

I do not know why your tests are not appearing. If you can provide more info, I can help you out. You should also try beta 2.

[h0useRus]

Thank you, great support, will check and wait MS guys to fix VS bugs. As I understand VS run first found framework of tests.

[psulek]

@rprouse Do you have link to ticket/issue for this bug in VS Net Core? To be able to track this error, thanks.

[clairernovotny]

Does it help to reverse the order of the frameworks in the project.json? I.e., have the netcoreapp1.0 one before the net451 one? In Rx.NET, I have both desktop and netcore target frameworks in my test project (with xunit) and it does show up in VS. Running on the command line executes both.

[rprouse]

@psulek I do no have the link for the bug in .NET Core. I was talking to the VSTest team last week and to @bradwilson from xUnit about it.

[ghost]

@h0useRus, if you have multiple frameworks in project.json, then the tests shouldn't disappear. Only the predicament is that we don't have explicit means of seeing which framework is currently selected by VS.

If your project is not public, then looking at your project.json, I can suggest few things:

Use this:

{
    "version": "1.2.3.4",

    "dependencies": {
        "MyLib": {
            "target": "project"
        },
        "NUnit": "3.2.1",
        "dotnet-test-nunit": "3.4.0-alpha-2"
    },

    "testRunner": "nunit",

    "frameworks": {
        "net451": { },       
        "netcoreapp1.0": {
            "imports": "portable-net45+win8",
            "dependencies": {
                "Microsoft.NETCore.App": {
                    "version": "1.0.0-*",
                    "type": "platform"
                }
            }
        }
    }
}

• recursive import to netcoreapp1.0 is unnecessary.
• dnxcore50 is a past history and deprecated.
• the only reason we need "imports": "portable-net45+win8", is because nunit supports portable profile (which works fine with netcoreapp on multiple platforms), but not explicitly support .NET Core yet.
• if you are using define: NETCORE to be able to have #if NETCORE pre-processor directive in your code, then use #if NETCOREAPP1_0 and no need for extra defines.

After that you can close VS, delete local packages directory (and %USERPROFILE%\.nuget to get a fresh start) and run dotnet restore --no-cache then dotnet test your/test/project/dir. Testing from VS at this point should also work, but it will only show test for one framework (first in the list). If you are still not seeing Tests in Test Explorer with more than one framework, then this is separate issue with Visual Studio. At least in community edition, it is showing the tests even when project.json has more than two frameworks specified.

[jcansdale]

I've been playing with a multi-framework project today and noticed that 'dotnet-test-nunit' doesn't currently work with classic .NET assemblies. I wonder if this could be related? Here is the repro:

project.json

{
  "version": "1.0.0-*",
  "buildOptions": {
    "emitEntryPoint": true
  },
  "frameworks": {
    "netcoreapp1.0": {
      "imports": "dnxcore50",
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.0"
        }
      }
    }
    ,
    "net451": {
      "buildOptions": {
        "define": [ "NET451" ]
      }
    }
  },
  "dependencies": {
    "NUnit": "3.4.1",
    "dotnet-test-nunit": "3.4.0-beta-1"
  },

  "testRunner": "nunit"
}

test.cs

namespace MultiTarget
{
    using NUnit.Framework;
    using System;

    public class Program
    {
        [Test]
        public static void Main()
        {
            Console.WriteLine("Hello, World!");
        }
    }
}

dotnet run -f net451

Project MultiTarget (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Hello, World!

dotnet test -f net451

Project MultiTarget (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
NUnit .NET Core Runner 3.4.0
Copyright (C) 2016 Charlie Poole
Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.10586
        Runtime: win10-x64
Test Files
    C:\Users\Jamie\Documents\Visual Studio 2015\Projects\MultiTarget\src\MultiTarget\bin\Debug\net451\win7-x64\MultiTarget.exe
System.DllNotFoundException: Unable to load DLL 'Microsoft.DiaSymReader.Native.amd64.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
   at Microsoft.Extensions.Testing.Abstractions.FullPdbReader.CreateSymReader64(Guid& id, Object& symReader)
   at Microsoft.Extensions.Testing.Abstractions.FullPdbReader.CreateNativeSymReader(Stream pdbStream)
   at Microsoft.Extensions.Testing.Abstractions.FullPdbReader..ctor(Stream pdbStream)
   at Microsoft.Extensions.Testing.Abstractions.PdbReaderFactory.Create(String pdbPath)
   at Microsoft.Extensions.Testing.Abstractions.SourceInformationProvider..ctor(String pdbPath, IPdbReaderFactory pdbReaderFactory)
   at NUnit.Runner.Navigation.NavigationDataProvider.GetNavigationDataProvider(String assemblyPath)
   at NUnit.Runner.TestRunner.Execute()
   at NUnit.Runner.TestRunner.Run(String[] args)

I'll investigate and report back.

[CharliePoole]

Huh! I wasn't aware it was supposed to work with classic .NET assemblies.

[ghost]

@jcansdale, while I can't reproduce this locally, #54 fixes this issue which was originally reported in #45.

@rprouse any ETA on next (beta) release? :)

[jcansdale]

#54 fixes this issue which was originally reported in #45.

Do you know what this fix does exactly? It seems strange for .NET 4.5.1 to be taking an dependency on a .NET Code thing. To be honest I don't understand what "Microsoft.NETCore.Platforms" is!

"net451": {
  "dependencies": {
    "Microsoft.NETCore.Platforms": "1.0.1"
  }
}

[ghost]

Do you know what this fix does exactly?

This is the CLI change; that it requires this package to resolve the dependency graph for transitive dependencies etc. The point of #54 is to prevent each consumer of the dotnet-test-nunit package to include this additional dependency, who is targeting net45x TxM.

[jcansdale]

@jasonwilliams200OK I'm afraid I didn't understand half of that, but I really want to! Could you unpack it a bit for me? I can confirm it does indeed fix it though. :)

[ghost]

@jcansdale, no problem, this is mostly undocumented. I actually went through couple of related issues in dotnet/cli and xunit repos to understand why dotnet test fails, in preparation for #54. Here is the most relevant bit https://github.com/dotnet/cli/issues/3016#issuecomment-225976094 which points to this blog for explaining the purpose of Microsoft.NETCore.Platform: http://www.natemcmaster.com/blog/2016/05/19/nuget3-rid-graph/.

The choice is to either include Microsoft.NETCore.Platform as dependency or add "runtimes" section with explicit list of RIDs, which is explained in this doc: https://docs.nuget.org/consume/projectjson-format (under Runtimes heading; no internal link on their site..).

The third option is to have a runtime.json file side-by-side our project.json to merely extract out the runtimes section in a separate file.

Since our library has no native-dependency and we want to support widest-possible audience, we want all permutations of platforms x RIDs to be included. To do that, the easiest way is to include Microsoft.NETCore.Platform (which just has a "runtime.json" file listing all the permutations for each supported OS, as we can find in this package's source: https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/runtime.json).

[deathrowe]

any update on the next beta release?

[rprouse]

I wasn't sure there was enough to warrant another release, but there have been a number of fixes. Let me look at what has been fixed and if there is enough, I will do a release this week.

@deathrowe which fixed issues are you interested in?

[deathrowe]

@rprouse specifically the fix from https://github.com/nunit/dotnet-test-nunit/issues/45

[rprouse]

@deathrowe that's a good one, I'll see what I can do :+1:

[ghost]

@deathrowe, FYI, beta-2 is released: http://www.nuget.org/packages/dotnet-test-nunit/3.4.0-beta-2

@h0useRus, we have updated the usage in README: https://github.com/nunit/dotnet-test-nunit#usage. Can you test if the bug reproduces with beta-2? If you can, please provide an SSCCE as I wasn't even able to reproduce it with beta-1 and VS2015 Update 3 Community Edition.