CTest integration not picking up test names on first compile

[reverendbedford]

Brief Issue Summary

I have a project where I am managing tests through Catch2. The CTest integration can successfully find, run, and display the results from my tests, but I'm facing an odd issue where it only works on the second compile of the project.

The first compile and run of Ctest finds no tests because instead of running the tests with a command of the form:

ctest -jxx -C Debug -T test --output-on-failure -R ^my_test_name-buildid

it instead runs

ctest -jxx -C Debug -T test --output-on-failure -R ^my_test_name_NOT_BUILT-buildid

The build itself actually looks fine, running ctest with no arguments in the build directory will run all the tests.

What's odd is that if I recompile (say by selecting a different build variant) and recompile/retest, changing nothing else, it will correctly pick up and run all the tests.

I can't figure out where the extension is picking up the test names, for example grepping for NOT_BUILT in the build directory returns nothing.

CMake Tools Diagnostics

No response

Debug Log

No response

Additional Information

No response

[benmcmorran]

Thanks for the report! I suspect the NOT_BUILT suffix is coming from Catch2, although I'm not familiar enough with Catch2 to understand why it does this. We will attempt to reproduce the issue to see if there's a way the CMake Tools extension can better handle this case and let you know if we need any more information.

[reverendbedford]

Thanks. Things do seem to work fine with the C++ TestMate extension, which is probably better integrated with Catch. It would be nice to be able to run the test properly from cmake-tools directly, but it's not a high priority if it doesn't happen.

[jeremicmilan]

I'm hitting the same issue but with GTest, but for me rebuilding is not helping. The fix for me is to refresh the tests (either through UI or with command cmake.refreshTests).

@benmcmorran, I have a similar file (<test_name>[1]_include.cmake) being generated in the build folder. Note that with GTest the suffix is slightly different.

if(EXISTS "<path>/<test_name>[1]_tests.cmake")
  include("<path>/<test_name>[1]_tests.cmake")
else()
  add_test(<test_name>_NOT_BUILT <test_name>_NOT_BUILT)
endif()

The file <path>/<test_name>[1]_tests.cmake does exist, and refresh is making the vscode integration pick it up.

A solid repro for me is to delete the build folder (so starting fresh) and clicking run all tests in the UI. So, I guess the fix is to run test refresh after the build finishes and prior to running the tests (but automatically).

[dcoles]

I'm hitting the same issue but with GTest, but for me rebuilding is not helping. The fix for me is to refresh the tests (either through UI or with command cmake.refreshTests).

I spent quite a bit of time debugging this today. From what I can tell it looks like VSCode is querying and caching the tests before the initial build is complete. As such the corresponding _tests.cmake file is not in the build tree and the test gets stubbed out with a _NOT_BUILT test.

So I agree that it would make sense to refresh the tests any time that build is run that touches a test target.

[v-frankwang]

@dcoles Thank you for the follow up. Our developers will be working on this issue, thank you for your patience.