Inconsistent behavior between ConsoleRunner 3.16.0 and 3.15.2

[goldcode]

Some tests of ours started failing after switching to 3.16.0. These tests don't fail via Visual Studio's Test Explorer or using Console Runner 3.15.2. I have added an example project here Calling the test assembly like so with different versions of the ConsoleRunner results in different results being returned.

c:\Users....nuget\packages\nunit.consolerunner\3.16.0\tools\nunit3-console.exe C:\Users...\source\repos\Test\ClassLibrary2\bin\Debug\net6.0\ClassLibrary2.dll

I'm uncertain ATM how the ConsoleRunner could influence the DeviceId library. Faulting code:

new DeviceIdBuilder()
                .OnWindows(windows => windows
                    .AddMotherboardSerialNumber()
                    .AddMacAddressFromWmi(excludeWireless: true, excludeNonPhysical: true))
                    .UseFormatter(new XmlDeviceIdFormatter(new PlainTextDeviceIdComponentEncoder()))
                    .ToString();

by chance, I figured out that if I extend the project file to include below, then the results are consistent again between both console runners.

  <PropertyGroup>
    <TargetFramework>net6.0-windows</TargetFramework>
    <OutputType>Library</OutputType>
    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
    <PlatformTarget>x64</PlatformTarget>
    <SelfContained>true</SelfContained>
    <IncludeAllContentForSelfExtract>true</IncludeAllContentForSelfExtract>
    <IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract>
    <EnableCompressionInSingleFile>true</EnableCompressionInSingleFile>
    <DebugType>embedded</DebugType>
    <DebugSymbols>true</DebugSymbols>
  </PropertyGroup>

[CharliePoole]

@goldcode The link to your example project doesn't work, so I'm not able to see what you specifically added to the project file in order to make it work. Also, are you running the console runner built for .NET Framework or the one for .NET 6.0? Either one should work but it makes life easier for me if I know which it is. :-)

[goldcode]

@CharliePoole the repo is now public.

I guess the console runner itself is running under 4.0 framework but targets assemblies for .NET 6.0. Is this the preferred approach?

log of 3.15.2

d:\Projekte\branches\trunk\Tools\Masterbuild\nunit.consolerunner>nunit3-console.exe c:\Users\xxx\source\repos\ClassLibrary2\ClassLibrary2\bin\Release\net6.0\ClassLibrary2.dll
NUnit Console 3.15.2 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Monday, 21 November 2022 20:37:16

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    c:\Users\xxx\source\repos\ClassLibrary2\ClassLibrary2\bin\Release\net6.0\ClassLibrary2.dll

<DeviceId><Component Name="MACAddress" Value="xxx" /><Component Name="MotherboardSerialNumber" Value="yyy" /></DeviceId>

Run Settings
    DisposeRunners: True
    WorkDirectory: d:\Projekte\branches\trunk\Tools\Masterbuild\nunit.consolerunner
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETCoreApp,Version=v6.0
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    TargetRuntimeFramework: netcore-6.0
    NumberOfTestWorkers: 24

Test Run Summary
  Overall result: Passed
  Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2022-11-21 19:37:17Z
    End time: 2022-11-21 19:37:17Z
    Duration: 0.729 seconds

Results (nunit3) saved as TestResult.xml

log of 3.16.0

d:\Projekte\branches\trunk\Tools\Masterbuild\nunit.consolerunner>c:\users\xxx\.nuget\packages\nunit.consolerunner\3.16.0\tools\nunit3-console.exe c:\Users\xxx\source\repos\ClassLibrary2\ClassLibrary2\bin\Release\net6.0\ClassLibrary2.dll
NUnit Console 3.16.0 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Monday, 21 November 2022 20:40:16

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    c:\Users\xxx\source\repos\ClassLibrary2\ClassLibrary2\bin\Release\net6.0\ClassLibrary2.dll

<DeviceId><Component Name="MACAddress" Value="xxx" /><Component Name="MotherboardSerialNumber" Value="" /></DeviceId>

Run Settings
    DisposeRunners: True
    WorkDirectory: d:\Projekte\branches\trunk\Tools\Masterbuild\nunit.consolerunner
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETCoreApp,Version=v6.0
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    TargetRuntimeFramework: netcore-6.0
    NumberOfTestWorkers: 24

Test Run Summary
  Overall result: Passed
  Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2022-11-21 19:40:17Z
    End time: 2022-11-21 19:40:17Z
    Duration: 0.488 seconds

Results (nunit3) saved as TestResult.xml

d:\Projekte\branches\trunk\Tools\Masterbuild\nunit.consolerunner>

[CharliePoole]

The NUnit.ConsoleRunner nuget package runner uses .NET Framework 4.6.2 (in 3.16) but launches a process for your .NET 6.0 executable. The NUnit.ConsoleRunner.NetCore package has a runner with fewer options, which runs under .NET 6.0 and executes the tests in the same process.

In both cases, this is a useful issue because it helps highlight the impact of various project properties.

[CharliePoole]

We're currently recommending that folks stop using version 3.16.x and use either 3.15.5 or 3.17.0. If the issue reappears in 3.17.0, we can reopen this issue.

[goldcode]

Thanks for the update. FYI, we use dotnet test meanwhile to run our nunit test assemblies. In the past we had a bunch of issues with nunit-console and reflection based code and had to perform some code acrobatics to get it to run due to the same module needing to be loaded twice in different context of the same process. Forgotten the details. And also because console app wasn't supporting wpf runtime.