Unable to set reporter when using ctest and catch_discover_tests

[sudara]

Describe the bug I'm unable to get any reporter other than console running using the latest catch v3 unless i'm explicitly passing the -r flag to the test binary.

Expected behavior/reproduction I expect to set the following in my CMakeLists.txt

catch_discover_tests(Tests REPORTER xml)

(where Tests is my tests target) then see xml output when going into my build folder and running

1. ./Tests Catch2 host application or
2. ctest

I'd love to actually see my reporter output in stdout when running ctest — ctest seems to generate entirely different output... Maybe my mental model is off, but seeing as we can define a reporter via catch_dicsover_tests, it seems like it should be used. However, ctest aside, it looks like the flag isn't making it to the Catch2 binary.

Platform information:

• OS: MacOS
• Compiler+version: Xcode 12.1
• Catch version: v3.0.0-preview.3

Additional context

./Tests --list-reporters

lists all reporters bundled with Catch2 as well as my custom reporter.

./Tests -r xml

Runs the correct reporter.

[horenmar]

Is the output you are seeing something like this?

26/34 Test #26: EscapeSpecialCharactersInTestNames ...................   Passed    0.15 sec
      Start 27: NegativeSpecNoHiddenTests
27/34 Test #27: NegativeSpecNoHiddenTests ............................   Passed    0.15 sec
      Start 28: TestsInFile::SimpleSpecs
28/34 Test #28: TestsInFile::SimpleSpecs .............................   Passed    0.17 sec
      Start 29: TestsInFile::EscapeSpecialCharacters
29/34 Test #29: TestsInFile::EscapeSpecialCharacters .................   Passed    0.16 sec
      Start 30: TestsInFile::InvalidTestNames-1
30/34 Test #30: TestsInFile::InvalidTestNames-1 ......................   Passed    0.17 sec

If so, then that is the standard CTest output. If you want to see what the underlying test wrote to stdout/stderr, you have to use either the --verbose flag

$ ctest -R RunTests --verbose
UpdateCTestConfiguration  from :/mnt/c/ubuntu/Catch2/clang-build/DartConfiguration.tcl
Parse Config file:/mnt/c/ubuntu/Catch2/clang-build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/mnt/c/ubuntu/Catch2/clang-build/DartConfiguration.tcl
Parse Config file:/mnt/c/ubuntu/Catch2/clang-build/DartConfiguration.tcl
Test project /mnt/c/ubuntu/Catch2/clang-build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: RunTests

1: Test command: /mnt/c/ubuntu/Catch2/clang-build/tests/SelfTest "--order" "rand" "--rng-seed" "time"
1: Test timeout computed to be: 1500
1: loose text artifact
1:
1: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1: SelfTest is a Catch v3.0.0-preview.3 host application.
1: Run with -? for options
1:
1: Randomness seeded to: 1613816859
1:
1: -------------------------------------------------------------------------------
1: run benchmark
1: -------------------------------------------------------------------------------
1: /mnt/c/ubuntu/Catch2/tests/SelfTest/IntrospectiveTests/InternalBenchmark.tests.cpp:401
1: ...............................................................................
1:
1: benchmark name                       samples       iterations    estimated
1:                                      mean          low mean      high mean
1:                                      std dev       low std dev   high std dev
1: -------------------------------------------------------------------------------
1: Test Benchmark                                 100            10    100.097 ms
1:                                           109.9 us      109.9 us      109.9 us
1:                                               0 ns          0 ns          0 ns
1:
1:
1: -------------------------------------------------------------------------------
1: mix info, unscoped info and warning
1: -------------------------------------------------------------------------------
1: /mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Message.tests.cpp:197
1: ...............................................................................
1:
1: /mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Message.tests.cpp:200: warning:
1:   and warn may mix
1:
1: /mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Message.tests.cpp:201: warning:
1:   they are not cleared after warnings
1:
1: ===============================================================================
1: All tests passed (2000 assertions in 282 test cases)
1:
1/1 Test #1: RunTests .........................   Passed    3.51 sec

The following tests passed:
        RunTests

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   3.63 sec

or even better, --output-on-failure, which only shows the output if the test failed.

[sudara]

Thanks @horenmar!

I understand that ctest overrides Catch output. This is what I'm trying to work around, so I can get at my reporter output. Running with ctest --verbose adds all kinds of noise before/after each test which renders it noisy and pretty useless for human parsing. Note that the test description shows up a total of 5 times here 😆

      Start 16: Some test description

16: Test command: /directory/Builds/Debug/Tests "Some test description" "--reporter clean"
16: Test timeout computed to be: 10000000
16: Some std out from another framework
16: Tests running Catch v3.0.0 with clean reporterFilters: Some test description
16: 
16: 
16:Some test description
16: ===============================================================================
16: ✅ All tests passed (1 assertion in 1 test case)
16: 
16/16 Test #16: Some test description .............   Passed    0.02 sec

I thought I'd try catch binary directly — however, the catch binary uses the default reporter unless I pass the -r on the command line to the binary, no matter how I try to set the reporter in the code or CMakeLists.txt.

Is this expected behavior and the only way to set a reporter is via a command line argument?

[b1ackviking]

The problem is that catch_discover_tests() generates a .cmake file with add_test() commands in the build directory with the following contents (an example from one of my projects):

add_test( [==[spi::Bus shows correct behavior]==] /home/b1ack/projects/onehome/build-ut/unit-test [==[spi::Bus shows correct behavior]==] [==[--reporter junit]==] )
set_tests_properties( [==[spi::Bus shows correct behavior]==] PROPERTIES WORKING_DIRECTORY /home/b1ack/projects/onehome/build-ut)
add_test( [==[spi::CC1352 shows correct behavior]==] /home/b1ack/projects/onehome/build-ut/unit-test [==[spi::CC1352 shows correct behavior]==] [==[--reporter junit]==] )
set_tests_properties( [==[spi::CC1352 shows correct behavior]==] PROPERTIES WORKING_DIRECTORY /home/b1ack/projects/onehome/build-ut)
set( unit-test_TESTS [==[spi::Bus shows correct behavior]==] [==[spi::CC1352 shows correct behavior]==])

As you can see, it sets the reporter to the one specified in the catch_discover_tests() call (junit in my case). However, the following snippet in the CatchAddTests.cmake comments it out:

https://github.com/catchorg/Catch2/blob/2cb5210caf35bf8fc29ade2e5570cc0f37537951/extras/CatchAddTests.cmake#L21-L28

[ahamez]

FWIW, I use the following shell script to list all tests and launch them with a reporter:

ctest --show-only=json-v1 | jq -r '.tests[] | .command[0]' | uniq  | xargs -I % sh -c '% --reporter JUnit::out=$(basename %).xml --reporter console::out=-::colour-mode=ansi'

[Hokbras]

The problem is that catch_discover_tests() generates a .cmake file with add_test() commands in the build directory with the following contents (an example from one of my projects):

add_test( [==[spi::Bus shows correct behavior]==] /home/b1ack/projects/onehome/build-ut/unit-test [==[spi::Bus shows correct behavior]==] [==[--reporter junit]==] )
set_tests_properties( [==[spi::Bus shows correct behavior]==] PROPERTIES WORKING_DIRECTORY /home/b1ack/projects/onehome/build-ut)
add_test( [==[spi::CC1352 shows correct behavior]==] /home/b1ack/projects/onehome/build-ut/unit-test [==[spi::CC1352 shows correct behavior]==] [==[--reporter junit]==] )
set_tests_properties( [==[spi::CC1352 shows correct behavior]==] PROPERTIES WORKING_DIRECTORY /home/b1ack/projects/onehome/build-ut)
set( unit-test_TESTS [==[spi::Bus shows correct behavior]==] [==[spi::CC1352 shows correct behavior]==])

As you can see, it sets the reporter to the one specified in the catch_discover_tests() call (junit in my case). However, the following snippet in the CatchAddTests.cmake comments it out:

https://github.com/catchorg/Catch2/blob/2cb5210caf35bf8fc29ade2e5570cc0f37537951/extras/CatchAddTests.cmake#L21-L28

Indeed, this snippet explains the issue. Thanks @b1ackviking

To quick and dirty fix that, this is what I've done:

• edit CatchAddTests.cmake, such as:

 foreach(_n RANGE 1 ${_last_arg}) 
   set(_arg "${ARGV${_n}}") 
+   if(_arg MATCHES "--reporter\ .*")
+     set(_args "${_args} ${_arg}")
+   elseif(_arg MATCHES "[^-./:a-zA-Z0-9_]") 
-   if(_arg MATCHES "[^-./:a-zA-Z0-9_]") 
     set(_args "${_args} [==[${_arg}]==]") # form a bracket_argument 
   else() 
     set(_args "${_args} ${_arg}") 
   endif() 
 endforeach() 

• in your CMakeLists.txt:

catch_discover_tests(Tests REPORTER xml
                                                OUTPUT_DIR "xml_reporter"
                                                OUTPUT_SUFFIX ".xml")

do note that you will have no more stdout feedback when your tests fail; everything will be reported in the .xml files generated: one per TEST_CASE.

[horenmar]

I looked through this and as far as I can tell there is no actual bug here for the current version of the CMake scripts (I did not check previous versions since this was opened). What is there is some confusion

• Setting REPORTER argument to catch_discover_tests is a CTest-only runtime thing. It has no effect on the actual compiled test binary, which will keep doing what it did before. CTest also does not default to passing through the output from the test, so with

catch_discover_tests(tests
  REPORTER xml
)

the short ctest output is, as expected,

    Start 1: abc
1/3 Test #1: abc ..............................   Passed    0.08 sec
    Start 2: def
2/3 Test #2: def ..............................   Passed    0.08 sec
    Start 3: ghi
3/3 Test #3: ghi ..............................   Passed    0.08 sec

The verbose CTest output is more like this

1: Test command: /mnt/c/ubuntu/temp/discover-tests-tests/build/tests "abc" "--reporter xml"
1: Working Directory: /mnt/c/ubuntu/temp/discover-tests-tests/build
1: Test timeout computed to be: 10000000
1: <?xml version="1.0" encoding="UTF-8"?>
1: <Catch2TestRun name="tests" rng-seed="2587399662" xml-format-version="3" catch2-version="3.5.2" filters="&quot;abc&quot;">
1:   <TestCase name="abc" filename="/mnt/c/ubuntu/temp/discover-tests-tests/tests.cpp" line="3">
1:     <OverallResult success="true" skips="0"/>
1:   </TestCase>
1:   <OverallResults successes="0" failures="0" expectedFailures="0" skips="0"/>
1:   <OverallResultsCases successes="1" failures="0" expectedFailures="0" skips="0"/>
1: </Catch2TestRun>
1/3 Test #1: abc ..............................   Passed    0.09 sec
test 2
    Start 2: def

...

this is noisy, but the XML reporter is properly used. If you also set up the OUTPUT_DIR and OUTPUT_SUFFIX (and any other related argument), you will properly get the output files in files as expected (these can then be fed into further tools to handle them).

Incidental protip for using CTest: put CTEST_OUTPUT_ON_FAILURE=1 into your environment, so that CTest gives short output for tests that passed and long output for tests that have failed.

• OP wanted to be able to set default reporter for the actual binary. This can be done via the CATCH_CONFIG_DEFAULT_REPORTER CMake option, but it must be set during compilation of Catch2 library.

• It would be nice to be able to set up multiple reporters for the CTest tests, because this is what the verbose output is when the XML reporter targets a file:

test 1
    Start 1: abc

1: Test command: /mnt/c/ubuntu/temp/discover-tests-tests/build/tests "abc" "--reporter xml" "--out /mnt/c/ubuntu/temp/discover-tests-tests/build/xml_reporter/abc.xml"
1: Working Directory: /mnt/c/ubuntu/temp/discover-tests-tests/build
1: Test timeout computed to be: 10000000
1/3 Test #1: abc ..............................   Passed    0.11 sec
test 2
    Start 2: def

2: Test command: /mnt/c/ubuntu/temp/discover-tests-tests/build/tests "def" "--reporter xml" "--out /mnt/c/ubuntu/temp/discover-tests-tests/build/xml_reporter/def.xml"
2: Working Directory: /mnt/c/ubuntu/temp/discover-tests-tests/build
2: Test timeout computed to be: 10000000
2/3 Test #2: def ..............................   Passed    0.11 sec
test 3
    Start 3: ghi

3: Test command: /mnt/c/ubuntu/temp/discover-tests-tests/build/tests "ghi" "--reporter xml" "--out /mnt/c/ubuntu/temp/discover-tests-tests/build/xml_reporter/ghi.xml"
3: Working Directory: /mnt/c/ubuntu/temp/discover-tests-tests/build
3: Test timeout computed to be: 10000000
3/3 Test #3: ghi ..............................***Failed    0.15 sec


67% tests passed, 1 tests failed out of 3

Total Test time (real) =   0.47 sec

The following tests FAILED:
          3 - ghi (Failed)
Errors while running CTest

notably there is absolutely no helpful output to figure out why ghi has failed without looking into the output files. This has its own issue in #2824