CTest (using catch_discover_tests) treats skipped tests as failed

[jonathanverner]

Describe the bug When using CMake integration to auto-discover tests and run them with CTest, tests which are skipped are reported as failed.

Expected behavior Skipped tests are reported as skipped.

Platform information:

• OS: Linux (Neon 22.04, based on Ubuntu 22.04)
• Compiler+version: 17.0.6
• Catch version: v3.3.2

Additional context I think the reason is that ctest runs the tests one-by-one and if a test is skipped, this actually means that no test was run so, since --allow-running-no-tests is not provided, the return code is non-zero.

Now, CTest allows marking skipped tests via a special return code by setting the SKIP_RETURN_CODE property on the test. So, it seems to me, this could be fixed by replacing the

set(properties ${TEST_PROPERTIES})

line in CatchAddTests.cmake with

set(properties "${TEST_PROPERTIES};SKIP_RETURN_CODE;4")

This works for me as does passing SKIP_RETURN_CODE 4 to catch_discover_tests PROPETIES argument, e.g. like so:

catch_discover_tests(middleware_tests
  PROPERTIES
    SKIP_RETURN_CODE 4
)

[fredemmott]

This ties into https://github.com/catchorg/Catch2/issues/1146 : unfortunately using SKIP_RETURN_CODE 4 means that if your test has 4 failed CHECK() invocations, CTest will consider it to be skipped.

[fredemmott]

For now I'm using SKIP_REGULAR_EXPRESSION "[1-9][0-9]* skipped"