Assert.Pass/Ignore/Inconclusive Skip Subsequent Steps In The Scenario

[Xen0byte]

SpecFlow Version

3.9.74

Which test runner are you using?

NUnit

Test Runner Version Number

3.9.74

.NET Implementation

.NET 7.0

Project Format of the SpecFlow project

SDK-style project format

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

Issue Description

When explicitly passing or ignoring a step or marking the step as inconclusive, the rest of the steps part of the same scenario are skipped.

Following is a real-world example use case of why this is a problem. I've built a framework that runs a set of back-end tests. Once a month, the golden copy of the data which get tested needs to be regenerated. To that purpose, in one of the feature files, there is a step along the lines of if this is the first test run of the current month, regenerate the golden copy so when this gets to execute it passes (for the sake of simplicity let's assume it never fails) but every other time when this step does not need to run then it does an Assert.Ignore("Golden Copy Regeneration Not Needed") and then all the next steps part of the same scenario get skipped. Currently the workaround is to use Assert.Warn() instead, but this creates inaccurate final test run results, because an outcome with warnings would be considered a soft failure.

Steps to Reproduce

Create a feature along the lines of ...

    Scenario:   [01/03] IGNORE
        Given   I IGNORE THE STEP
        When    I IGNORE THE STEP
        Then    I IGNORE THE STEP


    Scenario:   [02/03] INCONCLUSIVE
        Given   I SET THE STEP INCONCLUSIVE
        When    I SET THE STEP INCONCLUSIVE
        Then    I SET THE STEP INCONCLUSIVE


    Scenario:   [03/03] PASS
        Given   I PASS THE STEP
        When    I PASS THE STEP
        Then    I PASS THE STEP

... with step definitions such as ...

    [Given, When, Then]
    public void I_IGNORE_THE_STEP()
        => Assert.Ignore("IGNORE");

    [Given, When, Then]
    public void I_SET_THE_STEP_INCONCLUSIVE()
        => Assert.Inconclusive("INCONCLUSIVE");

    [Given, When, Then]
    public void I_PASS_THE_STEP()
        => Assert.Pass("PASS");

... and observe that step 1 passes, while steps 2 and 3 get skipped.

NOTE 1: The issue does not reproduce for Assert.Warn() or implicitly passed assertions (e.g. Assert.IsNull(null)). NOTE 2: The behaviour described above also reproduces for Assert.Fail(), however this is expected and should not change.

[SabotageAndi]

The output you posted is from a different scenario than you describe before. In the screenshot you have a "Given I ignore the step" written, what causes and error and because of that the rest of the steps are not executed.

[Xen0byte]

The output you posted is from a different scenario than you describe before. In the screenshot you have a "Given I ignore the step" written, what causes and error and because of that the rest of the steps are not executed.

The intention was to offer an example of both, but if that's misleading I will update the issue.

EDIT: Updated description with all 3 examples.