vscode-cmake-tools icon indicating copy to clipboard operation
vscode-cmake-tools copied to clipboard

[Bug] many ctest processes are launched freezing the system

Open JVApen opened this issue 6 months ago • 12 comments

Brief Issue Summary

I've noticed that on my system with 6 cores, there are more than 100 ctest executables running at the same time, causing the system to freeze. Relevant settings:

cmake.parallelJobs: 0 # Should adapt automatically to the cores of the system (aka 6)
cmake.ctest.parallelJobs: 0 # Should take over the cores of the previous option (aka 6)
cmake.ctest.allowParallelJobs: true # Enable parallel ctest
cmake.enableTraceLogging: true # Expect info in output

CMake Tools Diagnostics

{
  "os": "win32",
  "vscodeVersion": "1.102.3",
  "cmtVersion": "1.22.6",
  "configurations": [
    {
      "folder": "e:\\dev\\repo",
      "cmakeVersion": "3.30.2",
      "configured": true,
      "generator": "Ninja",
      "usesPresets": true,
      "compilers": {}
    }
  ],
  "cpptoolsIntegration": {
    "isReady": false,
    "hasCodeModel": false,
    "activeBuildType": "",
    "buildTypesSeen": [],
    "requests": [],
    "responses": [],
    "partialMatches": [],
    "targetCount": 0,
    "executablesCount": 0,
    "librariesCount": 0,
    "targets": []
  },
  "settings": [
    {
      "communicationMode": "automatic",
      "useCMakePresets": "auto",
      "configureOnOpen": true
    }
  ]
}

Debug Log

[main] Configuring project: main 
[main] Saving open files before configure/build

Occurs as many times as files where touched

In between it mentions:

[driver] Start configure 
[driver] Running pre-configure checks and steps
[cmakefileapi-driver] Configuring using preset
[cmakefileapi-driver] Invoking CMake C:/path/to/3.30.2/bin/cmake.exe with arguments [...]
[proc] Executing command: C:/path/to/3.30.2/bin/cmake.exe ...
[proc]   with environment: ...

Additional Information

Actions taken:

  • Find-replace some text in many CMakeLists.txt via VS Code find-replace
  • Dumped all diagnostics to a log file and searched for 'ctest', only found: 2025-08-20 11:23:31.128 [error] [Window] [Extension Host] [CMakeTools] 2025-08-20T09:23:31.127Z [error] [ctest] There was an error running ctest to determine available test executables
  • Checked the CMake/Build window, no mention of ctest

Expected behavior:

  • At most 6 ctest processes are running at the same time on a system with 6 cores

JVApen avatar Aug 20 '25 10:08 JVApen

Hi @JVApen , thank you for reporting this issue. We tried to check this issue in Visual Studio Code v1.103.1 + CMake Tools v1.22.6. But we can't reproduce this issue. Could you help to take a look at it? If we missed something, please let us know. We couldn't find any open source projects that include more CTest. In order for us to investigate this further, could you please provide a sample project to us that can reproduce this issue? Thanks.

repro.zip

yanghhhhhhh avatar Aug 21 '25 10:08 yanghhhhhhh

@yanghhhhhhh Do you have any tips on how to extract more information?

JVApen avatar Aug 21 '25 21:08 JVApen

@JVApen Thank you for your reply. Could you provide a sample project so that we can reproduce the issue? Could you collect a dump file? Thank you.

yanghhhhhhh avatar Aug 22 '25 09:08 yanghhhhhhh

@yanghhhhhhh I think we are running in circles. In order for me to extract a sample project out of hundreds of CMakeLists, I'm going to need an idea of what is relevant. Do you have any option that I can enable which lots better what it does? Can you add logging whenever such a process is launched, such that I can provide you with that info. Every attempt of reproducing this causes my system to be unresponsive for 15 minutes, so I can't just make a sample project out of this

JVApen avatar Aug 22 '25 18:08 JVApen

@JVApen Thanks for the information.

@gcampbell-msft Could you please take a look at this issue? Thanks.

yanghhhhhhh avatar Aug 26 '25 01:08 yanghhhhhhh

@JVApen Thanks for the issue.

To make sure I understand the issue, I want to make sure that there are a couple of things clear:

  1. I assume that this is happening when you attempt to run the tests?
  2. This could have to do with whether you have the parallel jobs settings, AS WELL as the test explorer integration enabled. What setting do you have for cmake.ctest.testExplorerIntegrationEnabled?
  3. From what I can tell, we are passing the -j parameter to specify the level of parallelism, which in the default case, should be os.cpus().length, using the node os.

Another thing that you can do, is to turn on the logging, I see you have the trace logging enabled, but please also set the cmake.loggingLevel to something like debug or trace. After doing this, does it give you more output?

gcampbell-msft avatar Aug 29 '25 11:08 gcampbell-msft

Extra info, I managed to get a hold of a command line of the ctest processes, they all contain the same:

C:\path\to\ctest.exe --show-only=json-v1 -T test --output-on-failure --progress --output-log ${sourceDir}/.out/testresults/results_${presetName}.log --output-junit E:/checkout/.out/testresults/results_custom_preset.xml --label-regex CUSTOM_LABEL --parallel 1 --timeout 600 --no-tests=error

Parent process is code.exe

I also realized that we are using set(CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE PRE_TEST), which probably will be related

JVApen avatar Sep 03 '25 08:09 JVApen

Regarding the questions:

  1. No, I'm not running any tests
  2. This has the default value, which is 'true'
  3. I'm not seeing the -j option, though that might be because this is not related to running the tests, it's related to discovery of them
  4. I've enabled 'trace', I'll come back once I have info based on that

JVApen avatar Sep 03 '25 08:09 JVApen

  1. I already did that when logging this issue, see what I wrote in 'Debug' in the original post. That's all the info that is printed.

JVApen avatar Sep 03 '25 08:09 JVApen

@gcampbell-msft : I hope this is sufficient for you

JVApen avatar Sep 03 '25 08:09 JVApen

@JVApen Is there are any way you can zip up a repro that we can test with?

Either way, some small investigation shows me the following:

It looks like you are using test presets, great! If you could send your test presets so that we can test with it, that'd be great. It seems like all of the options for the ctest invocations are being constructed based on this preset here: https://github.com/microsoft/vscode-cmake-tools/blob/fc290e4d4c3c21877666ab7b9d3a639717a4f93c/src/presets/preset.ts#L2343C1-L2411C2

The only places where we execute ctest are in the following locations:

  1. runCTest https://github.com/microsoft/vscode-cmake-tools/blob/fc290e4d4c3c21877666ab7b9d3a639717a4f93c/src/ctest.ts#L334C5-L371C6
  2. refreshTests https://github.com/microsoft/vscode-cmake-tools/blob/fc290e4d4c3c21877666ab7b9d3a639717a4f93c/src/ctest.ts#L890C4-L939C6

The first is only when we are executing tests, and you said that you're not executing tests. The second happens in a couple places, first, when a configure or build completes. Secondly, before executing tests, to make sure we have the full set. Neither of these should be causing lots of ctest.exe processes. Is there a relationship between the # of tests you have and the # of ctest processes?

I'll try to make a minimal repro and see if I can replicate, but from a code scan this is very odd and shouldn't be happening, so a zipped up project that is reproducing on your end would be most helpful. Thanks!

gcampbell-msft avatar Sep 08 '25 12:09 gcampbell-msft

Is there anything in your project that changes automatically the ctest settings within the cmake vscode settings? We automatically refresh the tests when that is changed. This is the only place in the code where I see something that may accidentally invoke a lot of ctest invocations

gcampbell-msft avatar Sep 08 '25 13:09 gcampbell-msft