CTest returns non-zero when any tests fail, causing coverage to not finish

[daviddoria]

As stated in the script:

# NOTE! This should always have a ZERO as exit code
# otherwise the coverage generation will not complete.

However, the recommended way of running coverage analysis on all files is to make an add_custom_target that runs ctest. If any tests fail, ctest returns non-zero which causes the coverage analysis to abort. It seems reasonable to still run coverage even if there are failing tests - is there any way to get around this?

[bilke]

Yes this can be a problem. Unfortunately add_custom_target always checks the exit code. Instead you could implement all COMMAND-steps in a CMake-script file which is generated at configure time and run at build time (from the docs):

To run a full script, use the configure_file() command or the file(GENERATE) command to create it, and then specify a COMMAND to launch it

Sorry I decided not to implement this because it adds a lot of complexity (functionality split into several files, generated script files in build-directory) and we run coverage only when the tests passed. I leave this issue open in case somebody else comes across this.

[daviddoria]

I guess I don't understand which add_custom_target is doing any exit code checking? In your script you just run COMMAND ${Coverage_EXECUTABLE}, so what difference does it make if that "fails" or succeeds? The files will still be there for the subsequent lcov calls to ingest, no?

[bilke]

In add_custom_target you can run several COMMANDs and if one of those return non-zero it simply stops execution of the remaining steps. There is no way around this. Another option would be to split into several add_custom_target-calls.

[daviddoria]

I ended up with the following that seems to work. Thanks for your help!

    add_custom_target(${Coverage_NAME}_cleanup
        COMMAND ${LCOV_PATH} --directory . --zerocounters
        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
        COMMENT "Old coverage cleanup..."
    )

    file(WRITE ${CMAKE_BINARY_DIR}/NoExitCodeTests.cmake "execute_process(COMMAND ctest)")

    add_custom_target(${Coverage_NAME}_runtests
        COMMAND ${CMAKE_COMMAND} -P NoExitCodeTests.cmake

        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
        COMMENT "Running tests..."
        VERBATIM
    )
    add_dependencies(${Coverage_NAME}_runtests ${Coverage_NAME}_cleanup)

    add_custom_target(${Coverage_NAME}
            # Capturing lcov counters and generating report
            COMMAND ${LCOV_PATH} --directory . --capture --output-file ${Coverage_NAME}.info
            COMMAND ${LCOV_PATH} --remove ${Coverage_NAME}.info ${COVERAGE_EXCLUDES} --output-file ${Coverage_NAME}.info.cleaned
            COMMAND ${GENHTML_PATH} -o ${Coverage_NAME} ${Coverage_NAME}.info.cleaned
            COMMAND ${CMAKE_COMMAND} -E remove ${Coverage_NAME}.info ${Coverage_NAME}.info.cleaned

            WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
            DEPENDS ${Coverage_DEPENDENCIES}
            COMMENT "Processing code coverage counters and generating report."
    )
    add_dependencies(${Coverage_NAME} ${Coverage_NAME}_runtests)

[arvedarved]

The proposed solution looks quite complicated, so I have used a wrapper

            setup_target_for_coverage(NAME coverage EXECUTABLE ${CMAKE_SOURCE_DIR}/ctestwrapper -j ${PROCESSOR_COUNT})

and the wrapper is just resetting the exit code:

#!/bin/sh
ctest $@
exit 0

[rgujju]

I faced the same issue and fixed it by changing COMMAND ${Coverage_EXECUTABLE} ${Coverage_EXECUTABLE_ARGS} to COMMAND ${Coverage_EXECUTABLE} ${Coverage_EXECUTABLE_ARGS} || exit 0 so even if ctest or any other test driver return non zero, the above command will always return 0.

Tested with cmake v3.13.3