nunit-console icon indicating copy to clipboard operation
nunit-console copied to clipboard
verified publisher icon nunit

Assembly loading move to AssemblyLoadContext for net8 and higher

Open SimonCropp opened this issue 2 weeks ago • 7 comments

Version 6 changes the assembly loading behavior to using the new AssemblyLoadContext when executing under net8 and higher.

This can have some subtle effects on how types are compared. For example if assemblies are being loaded via Assembly.LoadFrom or AssemblyLoadContext.Default then those assemblies (and the contained types) will not be comparable to the assemblies and types loaded using nunits AssemblyLoadContext.

This will often manifest in plugin based frameworks where assemblies on disk as scanned for known types

For example the below will pass on version 5.2 but fail on version 6.0

var assembly = Assembly.LoadFrom(file);
var type = assembly.GetType("Class1");
ClassicAssert.IsTrue(typeof(Class1) == type);

The fix is to move assembly loading over to using either:

  • nunits AssemblyLoadContext:
// can be cached statically
var context = AssemblyLoadContext.GetLoadContext(Assembly.GetExecutingAssembly())!;
var assembly = context.LoadFromAssemblyPath(file);

var name = AssemblyName.GetAssemblyName(file);
var assembly = Assembly.Load(name);

Tests

//All these work on V5.2
[TestFixture]
public class Tests
{
    string file = Path.GetFullPath("ClassLibrary.dll");

    //fails on 6.0
    [Test]
    public void WithLoadFrom()
    {
        var assembly = Assembly.LoadFrom(file);
        var type = assembly.GetType("Class1");
        ClassicAssert.IsTrue(typeof(Class1) == type);
    }

    [Test]
    public void WithGetAssemblyName()
    {
        var name = AssemblyName.GetAssemblyName(file);
        var assembly = Assembly.Load(name);
        var type = assembly.GetType("Class1");
        ClassicAssert.IsTrue(typeof(Class1) == type);
    }

    [Test]
    public void WithGetLoadContext()
    {
        var context = AssemblyLoadContext.GetLoadContext(Assembly.GetExecutingAssembly())!;
        var assembly = context.LoadFromAssemblyPath(file);
        var type = assembly.GetType("Class1");
        ClassicAssert.IsTrue(typeof(Class1) == type);
    }

    //fails on 6.0
    [Test]
    public void WithDefaultContext()
    {
        var context = AssemblyLoadContext.Default!;
        var assembly = context.LoadFromAssemblyPath(file);
        var type = assembly.GetType("Class1");
        ClassicAssert.IsTrue(typeof(Class1) == type);
    }
}

Full Repro

https://github.com/SimonCropp/NunitAdaptorAssemblyLoadingRepro

SimonCropp avatar Dec 08 '25 22:12 SimonCropp

my assumption is that this is an undocumented breaking change. so i am not expecting a fix.

this issue is intended as a reference issue for the release notes.

corresponding release notes PR: https://github.com/nunit/docs/pull/1116

if my assumption is incorrect, and a code change can be made to restore the v5 behavior, then this can be re-purposed or closed

SimonCropp avatar Dec 08 '25 22:12 SimonCropp

Thanks @SimonCropp !

@CharliePoole I assume this change is in the Engine, since we now use version 3.21.0.

Adapter version 4.6.0 - 5.2.0 used engine 3.18.1

OsirisTerje avatar Dec 09 '25 08:12 OsirisTerje

@OsirisTerje This is essentially the behavior we observed with nunit/nunit3-vs-adapter#1348, which was closed without action. AwsomeAssertions was attempting to load assemblies in the default AssemlyLoadContext, contrary to the engine's requirement of loading it in our own AssemblyLoadContext. This isn't a behavior we can readily change in V3, which allows direct user control over whether or not a separate process (also, for .NET Framework a separate AppDomain) in running tests. I'd like to revisit this in V4, so I'll transfer this one to the engine and tag it as V4.

@SimonCropp Your four test cases make a nice summary of the current status, I'll add them to my tests.

@OsirisTerje One ironic note, since this is arising in the adapter, is that we could probably safely use the default context in that case, since we believe that VS takes care of loading things for us anyway. We may want to experiment with this approach or even with not using the engine at all for your V7.

CharliePoole avatar Dec 09 '25 10:12 CharliePoole

I'm coming here after trying to upgrade NUnit3TestAdapter to v6. Below are some test cases that are passed for me when using NUnit3TestAdapter v5, but are now failing for me with NUnit3TestAdapter v6. I'm trying to use CSScript and I'm not sure of a workaround. Is this not considered a bug?

using CSScriptLib;
using Microsoft.CodeAnalysis.CSharp.Scripting;
using Microsoft.CodeAnalysis.Scripting;

namespace TestCase;

public class Point
{
    public int X { get; set; }
    public int Y { get; set; }
}

[TestFixture]
public class DynamicEvaluationTests
{
    [Test]
    public void TestCSScript()
    {
        dynamic script = CSScript.Evaluator.LoadCode(
            @"using TestCase;
             public class Script
             {
                 public Point CreatePoint(int x, int y)
                 {
                     return new Point { X = x, Y = y };
                 }
             }"
        );
        Point result = script.CreatePoint(1, 2);
        result.X.ShouldBe(1);
        result.Y.ShouldBe(2);
    }

    [Test]
    public async Task TestRoslyn()
    {
        var result = await CSharpScript.EvaluateAsync<Point>(
            "new Point { X = 1, Y = 2 }",
            ScriptOptions.Default.WithReferences(typeof(Point).Assembly).WithImports("TestCase")
        );
        result.X.ShouldBe(1);
        result.Y.ShouldBe(2);
    }
}
`TestCSScript` exception 
Microsoft.CSharp.RuntimeBinder.RuntimeBinderException : Cannot implicitly convert type 'TestCase.Point' to 'TestCase.Point'
at CallSite.Target(Closure, CallSite, Object)
at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
at TestCase.DynamicEvaluationTests.TestCSScript() (file:///Users/nathanbierema/Documents/ParagonCore/servers/libraries/GeometryLibrary/GeometryLibraryTests/LoggingTests.cs#L29,0)
at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
at CallSite.Target(Closure, CallSite, Object)
at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
at TestCase.DynamicEvaluationTests.TestCSScript() (file:///Users/nathanbierema/Documents/ParagonCore/servers/libraries/GeometryLibrary/GeometryLibraryTests/LoggingTests.cs#L29,0)
at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
`TestRoslyn` exception 
System.ArgumentException : Cannot bind to the target method because its signature is not compatible with that of the delegate type.
at System.Reflection.RuntimeMethodInfo.CreateDelegateInternal(Type delegateType, Object firstArgument, DelegateBindingFlags bindingFlags)
at System.Reflection.MethodInfo.CreateDelegate[T]()
at Microsoft.CodeAnalysis.Scripting.ScriptBuilder.Build[T](Compilation compilation, DiagnosticBag diagnostics, Boolean emitDebugInformation, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.ScriptBuilder.CreateExecutor[T](ScriptCompiler compiler, Compilation compilation, Boolean emitDebugInformation, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.GetExecutor(CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.RunAsync(Object globals, Func`2 catchException, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.RunAsync(Object globals, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.RunAsync[T](String code, ScriptOptions options, Object globals, Type globalsType, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.EvaluateAsync[T](String code, ScriptOptions options, Object globals, Type globalsType, CancellationToken cancellationToken)
at TestCase.DynamicEvaluationTests.TestRoslyn() (file:///Users/nathanbierema/Documents/ParagonCore/servers/libraries/GeometryLibrary/GeometryLibraryTests/LoggingTests.cs#L37,0)
at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.GetResult()
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func`1 invoke)
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func`1 invoke)
at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
at NUnit.Framework.Internal.Execution.SimpleWorkItem.<>c__DisplayClass3_0.<PerformWork>b__0()
at NUnit.Framework.Internal.ContextUtils.<>c__DisplayClass1_0`1.<DoIsolated>b__0(Object _)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at NUnit.Framework.Internal.ContextUtils.DoIsolated(ContextCallback callback, Object state)
at NUnit.Framework.Internal.ContextUtils.DoIsolated[T](Func`1 func)
at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()
at System.Reflection.RuntimeMethodInfo.CreateDelegateInternal(Type delegateType, Object firstArgument, DelegateBindingFlags bindingFlags)
at System.Reflection.MethodInfo.CreateDelegate[T]()
at Microsoft.CodeAnalysis.Scripting.ScriptBuilder.Build[T](Compilation compilation, DiagnosticBag diagnostics, Boolean emitDebugInformation, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.ScriptBuilder.CreateExecutor[T](ScriptCompiler compiler, Compilation compilation, Boolean emitDebugInformation, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.GetExecutor(CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.RunAsync(Object globals, Func`2 catchException, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.RunAsync(Object globals, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.RunAsync[T](String code, ScriptOptions options, Object globals, Type globalsType, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.EvaluateAsync[T](String code, ScriptOptions options, Object globals, Type globalsType, CancellationToken cancellationToken)
at TestCase.DynamicEvaluationTests.TestRoslyn() (file:///Users/nathanbierema/Documents/ParagonCore/servers/libraries/GeometryLibrary/GeometryLibraryTests/LoggingTests.cs#L37,0)
at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.GetResult()
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func`1 invoke)
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func`1 invoke)
at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
at NUnit.Framework.Internal.Execution.SimpleWorkItem.<>c__DisplayClass3_0.<PerformWork>b__0()
at NUnit.Framework.Internal.ContextUtils.<>c__DisplayClass1_0`1.<DoIsolated>b__0(Object _)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at NUnit.Framework.Internal.ContextUtils.DoIsolated(ContextCallback callback, Object state)
at NUnit.Framework.Internal.ContextUtils.DoIsolated[T](Func`1 func)
at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()

Methuselah96 avatar Dec 11 '25 01:12 Methuselah96

@Methuselah96 In order to give your report the fullest attention, it will be helpful if you can report it separately, rather than piggybacking on a different bug with somewhat different circumstances. Here are a few guidelines for doing so...

If the bug occurs when using the console runner and/or engine 3.21.0 directly, report it in this repo. If it works when using the engine directly but not under NUnit3TestAdapter, then report it there.

Since you are using dynamic script evaluation, which is not something I've seen used in tests before, please indicate where it has worked before, i.e. what versions of the tools, to give us a starting point. This will be useful whether you report it as an engine bug or an adapter bug.

Your examples are great and will help either team in working on the issue.

CharliePoole avatar Dec 11 '25 02:12 CharliePoole

Okay, thanks, I've never used NUnit.Console and I'm getting different exceptions when I try to do so, so I opened an issue in the adapter repo here, although I still suspect it's related. For context, I piggybacked here since my issue seemed most similar to this closed issue and I didn't want to create a bunch of new issues if they were all likely related.

Methuselah96 avatar Dec 11 '25 03:12 Methuselah96

@Methuselah96 Don't worry about creating new issues. It is easier for us to organize when they are separate. Added this one to a parent issue under the adapter, so just create your own, and since you have repro code, can you create a repro project to go with it, and upload that - we have one more thing to check :-)

OsirisTerje avatar Dec 11 '25 08:12 OsirisTerje

@SimonCropp @OsirisTerje

Closing this as fixed in master, MyGet version 3.21.1-alpha.2. The following is an explanation of what happened, what's fixed and what's (possibly) not fixed.

  1. All four tests worked under adapter 5.20 / engine 3.18.1 because both assemblies were always in the default context.
  2. Under 6.0.0 / 3.21.0 some tests failed because one assembly was in the default context while the other was in the TestAssemblyLoadContext.
  3. In this fix, I initially tried to use AssemblyLoadContext.Default but was unable to keep our regression tests running. Instead I switched to use of an instance of AssemblyLoadContext while continuing to apply the same resolution logic. This made three of the four test cases provided by @SimonCropp pass.
  4. The WithGetLoadContext failed mysteriously even though Intellisense shows the two assemblies as equal. By adding some Write statements, I discovered that the assembly is being loaded twice .

From this, I have learned two things...

  1. An assembly in a separate instance of AssemblyLoadContext may be "equal" to one in the default context.
  2. It's actually possible (at least under .NET 10.0) to load the same assembly twice in the same AssemblyLoadContext. Those two assemblies are not "equal", however. So doing this is probably a very bad idea. :-)

@SimonCropp I'm not 100% satisfied with this because I have not been able to reproduce the issue outside of the adapter. Ideally, we should have a test case that uses the engine in it's natural environment, i.e. without the adapter being involved. Something that runs under the console runner would be ideal, should you have the time.

CharliePoole avatar Dec 14 '25 23:12 CharliePoole