Missing test output from `[BeforeTestRun]` when using NUnit

[tbaliukynas]

Reqnroll Version

2.4.1

Which test runner are you using?

NUnit

Test Runner Version Number

4.3.2

.NET Implementation

.NET 8.0

Test Execution Method

Visual Studio Test Explorer

Content of reqnroll.json configuration file

No response

Issue Description

When running tests with the NUnit test runner, messages written to the standard output from within [BeforeTestRun] hooks are nowhere to be found. This is not the case when using xUnit or MSTest.

Steps to Reproduce

1. Create a new .NET test project:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />
    <PackageReference Include="Reqnroll.NUnit" Version="2.4.1" />
    <PackageReference Include="nunit" Version="4.3.2" />
    <PackageReference Include="NUnit3TestAdapter" Version="5.0.0" />
  </ItemGroup>
</Project>

2. Create a simple test and bindings that use System.Console for test output:

Feature: Test Output

Scenario: Writing to console
	Then write to console

[Binding]
public sealed class TestBinding
{
    [Then("write to console")]
    public void ThenWriteToConsole() => Console.WriteLine(nameof(ThenWriteToConsole));

    [BeforeTestRun]
    public static void BeforeTestRun() => Console.WriteLine(nameof(BeforeTestRun));

    [BeforeFeature]
    public static void BeforeFeature() => Console.WriteLine(nameof(BeforeFeature));

    [BeforeScenario]
    public void BeforeScenario() => Console.WriteLine(nameof(BeforeScenario));
}

3. Notice that the BeforeTestRun message is missing from the standard output in the Test Detail Summary pane within the Test Explorer window, as well as from the Output window:

========== Starting test run ==========
NUnit Adapter 5.0.0.0: Test execution started
Running selected tests in C:\Repos\ReqnrollProject\ReqnrollProject\bin\Debug\net8.0\ReqnrollProject.dll
   NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run
BeforeFeature
BeforeScenario
Then write to console
ThenWriteToConsole
-> done: TestBinding.ThenWriteToConsole() (0.0s)

NUnit Adapter 5.0.0.0: Test execution complete
========== Test run finished: 1 Tests (1 Passed, 0 Failed, 0 Skipped) run in 794 ms ==========

Link to Repro Project

No response

[304NotModified]

According to copilot this could be a nunit issue. I'm not sure if that is correct (or made up), but it could be good to check.

[gasparnagy]

Yes, this is an NUnit issue. @tbaliukynas You can try to run the tests as dotnet test --logger "console;verbosity=detailed"

[Chembeti]

I am also facing the same issue (using the same package versions mentioned in the bug). My [BeforeTestRun] method is not called at all.

[Navis62]

Same here

[304NotModified]

Have someone tried this? https://github.com/reqnroll/Reqnroll/issues/670#issuecomment-3055963442

If that won't work, could someone please create a small repo case? Thanks!

[CaptainKernow]

I am also seeing this issue with NUnit

Steps to replicate:

1. Create new .net Reqnroll test project

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.10.0" />
    <PackageReference Include="Reqnroll.NUnit" Version="2.2.1" />
    <PackageReference Include="nunit" Version="3.14.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
  </ItemGroup>

  <ItemGroup>
    <Folder Include="Support\" />
  </ItemGroup>

</Project>

2. Use default Test and create Bindings that output to console

Scenario: Add two numbers
	Given the first number is 50
	And the second number is 70
	When the two numbers are added
	Then the result should be 120

namespace ReqnrollTestProject.Drivers
{
    [Binding]
    public class Hooks
    {
        [BeforeTestRun]
        public static void BeforeTestRun()
        {
            Console.WriteLine("++++Before Test Run output");
        }

        [BeforeScenario]
        public void BeforeScenario()
        {
            Console.WriteLine("****Before Scenario output");
        }

        [AfterScenario]
        public void AfterScenario()
        {
            Console.WriteLine("****After Scenario output");
        }

        [AfterTestRun]
        public static void AfterTestRun()
        {
            Console.WriteLine("++++After Test Run output");
        }
    }
}

3. Run Tests with dotnet test --logger "console;verbosity=detailed"

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1.2363 Seconds
  ReqnrollTestProject test succeeded with 1 warning(s) (1.5s)
    C:\Program Files\dotnet\sdk\9.0.305\Microsoft.TestPlatform.targets(48,5): warning :
      ****Before Scenario output
      Given the first number is 50
      First number: 50
      -> done: CalculatorStepDefinitions.GivenTheFirstNumberIs(50) (0.0s)
      And the second number is 70
      Second number: 70
      -> done: CalculatorStepDefinitions.GivenTheSecondNumberIs(70) (0.0s)
      When the two numbers are added
      Adding the two numbers
      -> done: CalculatorStepDefinitions.WhenTheTwoNumbersAreAdded() (0.0s)
      Then the result should be 120
      Result: 120
      -> done: CalculatorStepDefinitions.ThenTheResultShouldBe(120) (0.0s)
      ****After Scenario output


Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 1.5s
Build succeeded with 1 warning(s) in 5.1s

Expected: "++++Before Test Run output" and "++++After Test Run output" should be present in the above log, indicating that [BeforeTestRun] and [AfterTestRun]

Actual: Only [BeforeScenario] and [AfterScenario] outputs are shown

Am happy to share a repo of this if needed

[gasparnagy]

I have also tried in the meantime, but I haven't found any way to provide an output even in a simple (non-Reqnroll) NUnit project... So we can't do much. You can write out your messages to a log file...

[johncmJanes]

FYI, I think you should be able to use this as a work around using NUnit's own TestContext class...

TestContext.Progress.WriteLine("Before test run");

[clrudolphi]

FYI, I think you should be able to use this as a work around using NUnit's own TestContext class...

TestContext.Progress.WriteLine("Before test run");

Frustrating results.

Using TestContext.Progress.WriteLine() DID result in output being written by [BeforeTestRun] and [AfterTestRun] hooks, but ONLY when executed from the command-line. Output from those two hooks is NOT displayed in the test log when the test is run by the VisualStudio2022 Test Explorer.

Solving this still seems out of our hands.