Run a single example

[provegard]

I'm trying to create a useful NUnit wrapper for NSpec. I have created an NUnit TestDataSource that discovers all NSpec examples and generates NUnit test cases on the fly. That part works.

However, I cannot run the examples during the discovery phase, because things like setup fixtures haven't run at that point. So in the actual "example realization" method, I want to run a single example.

My current approach involves wrapping the context of the example in a ContextCollection, but the problem is that the Context contains all examples, so I observe the same example being run multiple times.

Unsuccessful attempts so far:

• Tweak each example by giving it a GUID tag, and then creating a tag filter with that GUID.
• Setting HasRun on each example so that only the one about to run has true.

I will post some code later, but perhaps you have some idea right away?

[amirrajan]

I'm trying to create a useful NUnit wrapper for NSpec.

I think @BrainCrumbz is working on a wrapper.

That being said, here is how NSpec executes specifications:

//types that should be considered for testing
var types = Assembly.GetExecutingAssembly().GetTypes();

//now that we have our types, set up a finder so that NSpec
//can determine the inheritance hierarchy
var finder = new SpecFinder(types);

//we've got our inheritance hierarchy,
//now we can build our test tree using default conventions
var builder = new ContextBuilder(finder, new DefaultConventions());

//create the nspec runner with a
//live formatter so we get console output
var runner = new ContextRunner(
         builder,
         new ConsoleFormatter(),
         false);

//create our final collection of concrete tests
var testCollection = builder.Contexts().Build();

//run the tests and get results (to do whatever you want with)
var results = runner.Run(testCollection);

//console write line to pause the exe
System.Console.ReadLine();

You could delete entries from the collection before var results = runner.Run(testCollection);.

Thoughts?

[BrainCrumbz]

We created a Visual Studio Test Adapter for NSpec, NSpec.VsAdapter, so that a developer can list, group, select and launch NSpec specs directly from within Visual Studio. Here's a (tiny) screenshot, but generally people used to Visual Studio know how that works.

Actually, I did not get what should be a NUnit wrapper for NSpec.

[provegard]

Sorry for the delay. I just sat down to create a GitHub repo to illustrate the problem, but I got momentarily stuck at some changes between NUnit 2 and 3.

Anyway, my use case is that I have a project with around 10000 NUnit tests. Changing those to NSpec isn't feasible, but I'd like to start using NSpec here and there because I like it. My two requirements are:

1. It should be possible to run NSpec examples using ReSharper.
2. The NSpec examples should be included when the test suite is run using the NUnit console runner.

To accomplish that, I started from DebuggerShim and created a base class that uses a NUnit TestCaseSource method to generate one NUnit test case for each discovered NSpec example.

My first version ran the exampes (like DebuggerShim does) from inside the TestCaseSource method, but it's a bad idea since that method is called too early in NUnit's life cycle - before important setup fixtures have run.

Thus, I split discovery and running of example-derived test cases. It almost works, but because ContextRunner runs a ContextCollection I have to give it the context of the example. But that context refers to all its examples, so the effect is that all examples will run as part of the first test case.

I'll get back with a repo that demonstrates the problem more clearly.

[provegard]

Ok, here's a demo repo:

https://github.com/provegard/NSpecInNUnit

(Open with VS2015, enable NuGet package restore.)

When I run the a_spec class using ReSharper, I get three failing tests. They fail because the runner expects each test to run a single example, but the first test runs all three examples, and the subsequent tests run zero examples each.

[amirrajan]

Can you walk me through how run the repo? I have building in VS 2015, what do I do next? Can I use TDD.Net? (don't have ReSharper). I can get a license if needed >_<

[provegard]

The point is that I use ReSharper or NUnit console runner. With ReSharper installed, you get an icon next to the test class that you can use to run the class:

But I have updated the repo with a reference to NUnit.Runners, so if you pull and then rebuild to let NuGet install the new package, you can then run the following from the Windows command prompt:

packages\NUnit.ConsoleRunner.3.2.0\tools\nunit3-console.exe NSpecInNUnit\bin\Debug\NSpecInNUnit.dll

Both approaches result in the same thing: Three failing tests.

[amirrajan]

Perfect, let me give it a shot and see what I find.

[amirrajan]

Extremely hokey, but I think this worked. We need to make a change in NSpec that completely decouples the Builder from the Runner. I can't think of a reason off the top of my head why Runner needs to take in a Builder. Committed it here

[TestCaseSource(nameof(Examples))]
public void RunExample(ExampleBase example, SpecFinder finder)
{
    // Unsuccessful attempt to run one example per test:
    // var aTag = Guid.NewGuid().ToString();
    // example.Tags.Add(aTag);
    // ...and then `new Tags().Parse(aTag)` below.

    // A dummy ContextBuilder to get Tags into the ContextRunner
    var builder = new ContextBuilder(finder, new Tags(), new DefaultConventions());
    var runner = new ContextRunner(builder, new NoopFormatter(), false);
    var testSuite = builder.Contexts().Build();

    testSuite.AllContexts().Do(c =>
    {
        c .AllExamples()
        .Where(e => e.FullName() != example.FullName())
        .Do(e =>
        {
            e.HasRun = true;
            e.Pending = true;
        });
    });

    runner.Run(testSuite);

    if (example.Failed())
    {
        if (example.Exception != null)
        {
            // Use ExceptionDispatchInfo to preserve the original stack trace
            var edi = ExceptionDispatchInfo.Capture(example.Exception);
            edi.Throw();
        }
        throw new Exception("It failed :-(");
    }
}

[BrainCrumbz]

In currently avaliable package version, 1.0.4, ContextRunner still requires a ContextBuilder (code). But since this (unpublished ?) commit that dependency has been removed.

The reason seemed to be that the Runner needed the Builder just to query it for Tags, which seemed a little convoluted and beyond Builder responsibilities. It seemed more appropriate for RunnerInvocation to pass Tags both to ContextBuilder as well as ContextRunner.

[amirrajan]

I can push an update of the Nuget package (if I'm reading what you're saying correctly). Do you want to merge in the "fail if async" stuff first?

[BrainCrumbz]

Yep, PR just merged

[amirrajan]

1.0.5 has been pushed to Nuget. @provegard, you may be able to go back to your original approach with the new update.

[provegard]

Thanks, the HasRun approach worked. I could swear I've tried something like that before (minus setting Pending) and without getting it to work.

@amirrajan I'm not sure what you mean with "original approach". I upgraded to 1.0.5 and it was nice to get rid of the dummy builder. :-)

[amirrajan]

I upgraded to 1.0.5 and it was nice to get rid of the dummy builder. :-)

You should be able to manipulate the context collection and remove examples all together as opposed to having to set HasRun and Pending to true. I may be mistaken on this however.

Anyways, really glad that it worked for you. Wonder if @BrainCrumbz VS plugin could benefit from your NUnit wrapper.

[BrainCrumbz]

@amirrajan I was going to close this issue for inactivity, but then there's quite a point in being able to run NSpec through a single spec/test-case, instead of a whole (sub)context as it is now.

Could you please elaborate further the reasoning started in previous comment ("We need to make a change in NSpec that completely decouples the Builder from the Runner."). Now that they are decoupled, what's after?

[amirrajan]

I think we're good. This commit decouples: https://github.com/mattflo/NSpec/commit/ccc5af0a78ebb55492f52762cab14528996c288e

what's after

Don't want to push this direction too hard, but I wander if there is a benefit to codify the entire test structure. It's kind of already there with the ContextCollection, just wondering if we can have better visibility into the order of execution instead of it being decided when the examples are exercised. Does that make sense?

To put it another way. The runner, just does a really dumb job and executes the examples and records exceptions. The builder "rolls out" all the before's, acts, etc, so you can see the exact order everything would execute in (before it executes).

[BrainCrumbz]

Not sure how such codification of entire test structure would help, our main concern was to provide an API into nspec to run one or more selected its, while now one can "only" run all its within a context. The final purpose would be to expose such an API to NSpec console runner and VS test adapter.

It seems that such an API is not really about exact order of execution, but more about:

1. how to specify one or more its to execute, potentially across several contexts (one can figure out this reasonably easy)
2. how to run just selected its in a context, without running all of them. Of course, the correct hooks should also run before and after (maybe this sounds somehow more complicated)

[amirrajan]

Gotcha. I think we've discussed this before, but I really do think that method level execution is sufficient (or at least a good starting point). Usually when the context nest too much, I "push everything up".

Not sure if you know this, but every test is Taged with the exact test name. So you could call the runner with that explicit test name and only that test will run.

Bad memory at this point, I'm sure we've talked through all of this before :-)

[BrainCrumbz]

I don't think I knew that there's a Tag for every it, while I was aware there's one for every nspec-derived class. It must be memory then :)

[amirrajan]

Ah, I think you're right. Think it'd be a good idea to add a tag for every it (or at least context)? Or would that be a terrible hack that would make baby Jesus weep?