Can't suppress headers and other output when executing tests

[jcansdale]

I've tried using the following switches: --labels=Off --trace=Off --noheader, but the output when running a single test seems quite verbose:

Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.14393
        Runtime: win10-x64

Test Files
    C:\Source\GitHub\NetCoreTestRunner\TestDriven.NetCore.TargetApplication\bin\Debug\netcoreapp1.0\TestDriven.NetCore.TargetApplication.dll

Test Filters
    Test: TestDriven.NetCore.TargetApplication.NUnitTests.TestClass

Passed

Here it is without any switches:

NUnit .NET Core Runner 3.4.0 
Copyright (C) 2016 Charlie Poole

Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.14393
        Runtime: win10-x64

Test Files
    C:\Source\GitHub\NetCoreTestRunner\TestDriven.NetCore.TargetApplication\bin\Debug\netcoreapp1.0\TestDriven.NetCore.TargetApplication.dll

Test Filters
    Test: TestDriven.NetCore.TargetApplication.NUnitTests.TestClass

=> TestDriven.NetCore.TargetApplication.NUnitTests.TestClass.Pass
Passed

1. Maybe the "Passed" output could be switched off with --labels=Off (it only really makes sense alongside the label).
2. The header includes NUnit .NET Core Runner 3.4.0, even when executing with the .NET Framework.

[rprouse]

I agree, suppressing headers doesn't trim it down much. We should decide what should be hidden and probably make the same changes in the console. What do you think minimal info should be?

[CharliePoole]

@rprouse We might want to add some further options as to what should be suppressed, but IMO --noheader should cut out the header, as it has always been defined, and not anything else.

Probably best to decide what the different levels should look like however.

@jcansdale The "Passed" seems to be output from your test. Shutting off labels means you don't want the output labeled, not to suppress the output itself. Of course, if I'm wrong and it's the runner producing "Passed" as part of the label, it's a bug.

[jcansdale]

@CharliePoole,

Here is the output from two tests using simply dotnet test.

C:\Source\Scratch\TestsXproj>dotnet test
Project TestsXproj (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
NUnit .NET Core Runner 3.4.0
Copyright (C) 2016 Charlie Poole
Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.14393
        Runtime: win10-x64
Test Files
    C:\Source\Scratch\TestsXproj\bin\Debug\netcoreapp1.0\TestsXproj.dll
=> TestsXproj.Tests.Test1
Passed
=> TestsXproj.Tests.Test2
Failed
Errors and Failures
1) Failed : TestsXproj.Tests.Test2
Boom!
at TestsXproj.Tests.Test2() in C:\Source\Scratch\TestsXproj\Tests.cs:line 16
Run Settings
    WorkDirectory: C:\Source\Scratch\TestsXproj
Test Run Summary
  Overall result: Failed
  Test Count: 2, Passed: 1, Failed: 1, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 1, Errors: 0, Invalid: 0
  Start time: 2016-10-04 18:56:19Z
    End time: 2016-10-04 18:56:20Z
    Duration: 0.526 seconds
Results saved as C:\Source\Scratch\TestsXproj\TestResult.xml
SUMMARY: Total: 1 targets, Passed: 0, Failed: 1.

I'm pretty sure it's the NUnit runner that outputs Passed and Failed.

[rprouse]

Very odd, I am not seeing the labels or the Pass/Fail when I run unit tests and I am pretty sure we are not outputting the pass/fail. Maybe it is because mine targets net461? I guess I will need to do some testing 😄

NUnit .NET Core Runner 3.4.0
Copyright (C) 2016 Charlie Poole
Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.14393
        Runtime: win10-x64
Test Files
    D:\src\web\xxx\test\FoodGrads.Test\bin\Debug\net461\win7-x64\xxx.Test.dll
Run Settings
    WorkDirectory: D:\src\web\xxx
    NumberOfTestWorkers: 8
Test Run Summary
  Overall result: Passed
  Test Count: 54, Passed: 54, Failed: 0, Inconclusive: 0, Skipped: 0
  Start time: 2016-10-04 19:30:51Z
    End time: 2016-10-04 19:30:54Z
    Duration: 2.515 seconds
Results saved as D:\src\web\xxx\TestResult.xml
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.

[jcansdale]

@rprouse,

Here's my project.json for reference:

{
  "version": "1.0.0-*",

  "dependencies": {
    "NUnit": "3.4.1",
    "dotnet-test-nunit": "3.4.0-beta-2"
  },

  "testRunner": "nunit",

  "frameworks": {
    "netcoreapp1.0": {
      "imports": "dnxcore50",
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.0"
        }
      }
    }
  }
}

[rprouse]

@jcansdale it looks like the Pass/Fail labels are coming from #56

[jcansdale]

Looks rather like it. :)

[BrainCrumbz]

We're experiencing the same, on dotnet-test-nunit 3.4.0-beta-3. With no specific option:

=> the full test case name Passed

With --labels Off:

Passed

Is this behaviour intended? Thanks!

[jskeet]

I've got exactly the same with 3.4.0-beta3, which wasn't the case for 3.4.0-beta1. This basically makes that build unusable for Noda Time, which has 15000 tests. Sample build log: https://travis-ci.org/nodatime/nodatime/builds/205057166

Happy to look into this if it would be useful. I don't know how much of this is going to change with the new .NET Core tooling though - I completely understand if it's not worth looking into right now.

[rprouse]

@jskeet it is all changing for the new .NET Core tooling, but I would be willing to put out one more release to hold people over, so if someone submits a PR, I would accept it.

[jskeet]

Thanks for the assessment. I'll see if I get time to investigate, although I suspect I won't. There's nothing I need from the newer version, so I'll be happy enough to wait for the new tooling.

Is #91 the best way of following the support for the new tooling, or is there an issue in a different project that's worth subscribing to?