Add option like xUnit's appDomain=denied

[mzboray]

I recently happened to need to use xUnit.net for a test project because it had one key feature that seemed to be missing from NUnit. This is an option to not create any appdomain's when running tests. xUnit.net supports this as part of its configuration (see appDomain=denied). I looked at several NUnit issues and Stack Overflow questions but could not find an exact equivalent to this option, at least for the VS adapter. I believe the NUnit console has its own runner options for this but haven't looked into it too closely.

There is the DisableAppDomain of runsettings however I could not get this to work for my use case. I believe NUnit and xUnit behave the same with this setting but it still seems to create an appdomain. (xUnit does not work as desired for me with this setting.)

I realize there might be some limitations when you do this, and I am willing to deal with them as long as this option exists. For example, even with the xUnit.net option the test author needs to take over loading any dependencies with AppDomain.AssemblyResolve since the primary appdomain is the startup directory of the testhost process.

For more specifics about my use case, I was trying to write some unit tests involving the CefSharp library, which has several strict requirements: must be initialized and shutdown in the same thread, can only be initialized in the primary/default appdomain. Testing this code with NUnit with Visual Studio integration wasn't possible. I suppose I could have removed the VS adapter and tried to run these tests with the console runner, but that was too much of a hassle to deal with for a large project, especially if it meant replumbing our test pipeline in Azure DevOps.

[OsirisTerje]

Can you provide a small repro with instructions to showcase this, and which we can use for verification?

[mzboray]

I've put an example in this repo comparing to xunit's behavior: https://github.com/mzboray/nunitappdomainexample

[mzboray]

I've expanded the example I had that was an extremely basic comparison between xunit and nunit's treatment of appdomains to a more fully realized demonstration that is more realistic.

[OsirisTerje]

Thanks :-) I'll have a look at it as soon as I can!

[ChrisMaddock]

@mzboray - could you try running this in the NUnit Console with --domain=None specified, and see if that works as required?

@OsirisTerje - if so, this would be another one the engine can provide for free, but just exposing the option. 🙂

[mzboray]

Yes. If you check out the repo I posted, build.ps1 will run the tests run successfully with the console runner and domain=None option. So my ask is really for this feature to be exposed in the vs adapter somehow. I guess xunit allows this to configured per project via app.config appsettings, if that makes any difference. Also it would help to clarify in some documentation what the runsettings DisableAppDomain is supposed to be used for. Thanks.

[ChrisMaddock]

Thanks @mzboray! Sorry, was browsing on my phone, didn’t realise you’d already found that. 😊

[m4ggo]

I had a similar issue and from what I observed the root cause is that once the 'DisableAppDomain' is set to 'true' in the .runsettings VS (with 2017 and 2022) already invokes the test adapter within a new AppDomain. The 'domain=None' option is recognized by the testadapter and passed to the engine but since the adapter itself is not in the default AppDomain it does not have the same behaviour as the console runner which is running the tests in the default AppDomain with the 'domain=None' argument. If the 'DisableAppDomain' is not set or false the adapter is invoked in the default AppDomain - so if the 'DomainUsage' setting would be exposed via a NUnit specific option it would fit the behaviour of the console runner. Exposing this option is straightforward I can provide a PR if this should be included.

[m4ggo]

Unfortunately just exposing the 'DomainUsage' setting of the adapter doesn't work for the 'None' case - the base path of the default AppDomain when executing the tests in VS is located somewhere in the VS installation directory. This causes the default assembly loading to break and would require a custom AssemblyResolver which locates at least the files in the directory of the test assembly.