nunit-console
nunit-console copied to clipboard
Create pseudo-runners for situations where we can't actually run tests but want to produce a report
BACKGROUND
When the runner or engine encounters a file name for which we can't run tests, we could obviously just throw an exception and terminate. That's acceptable if the invalid file is referenced on the console runner command-line. It's not OK in several other situations:
- When a runner with a continuous UI display (GUI or otherwise) needs to get the information about an invalid file and display it. It can't do that if it terminates.
- The invalid file may be nested inside a project containing many other files. We may want to run the tests in the valid files and just display an error for the invalid files.
CURRENT HANDLING
We currently use special drivers in the engine to handle this. They all derive from the abstract class NotRunnableFrameworkDriver
. Separate subclasses exist for invalid assemblies and assemblies, which we are skipping because they don't contain any tests.
DRAWBACKS OF THE CURRENT APPROACH
- Since we use drivers, we need to create an agent process in many cases. This is costly, since these drivers don't actually execute any code in the tests but merely report back a result in the correct format.
- Issue #1468 pointed out to me that this approach doesn't work in all situations. That issue relates to unmanaged executables, which cannot be loaded by any of our agents.
PLANNED APPROACH
I plan to create a set of NotRunnableTestRunners
to replace the drivers. The UnmanagedExecutableTestRunner
I created for #1468 will serve as a prototype. Each existing NotRunnableFrameworkDriver
, will e sysetmatically replaced by a NotRunnableTestRunner
.
These runners (or pseudo-runners) will not follow the ProcessModel
and DomainUsage
settings used by all runnable test files. They will execute immediately, in process. None of the drivers actually examine the files whose names are passed to them because that has already been done when they are activated. That will remain true for the runners.
Suggestions or Comments? @nunit