CMake Testing - CMake presets not used correctly?

[nikolaseu]

Brief Issue Summary

I am using CMakePresets.json to include some extra necessary folders in the PATH. I also use google test gtest_discover_tests(). I can build and run tests successfully from the command line (using CMake's --preset).

But when I try to build from vscode, it fails. The command executed by vscode is different, it's not including the --preset argument, even when I have correctly selected and configured with the right preset (the build preset is also correctly selected). It seems to parse the preset file and add the arguments manually? Why is that?

In my case the build fails because gtest_discover_tests() cannot run the executable (to discover the tests) because the required DLLs are not in the PATH:

[build] CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.28/Modules/GoogleTestAddTests.cmake:112 (message):
[build]   Error running test executable.
[build] 
[build]     Path: 'C:/******/build/x64-release/bin/tst_***.exe'
[build]     Result: Exit code 0xc0000135

In addition, if I configure and build correctly from the command line (using --preset), then running the tests from the "Testing" panel also fail with Exit code 0xc0000135, the ctest command is again not including the --preset argument, so the environment variables are not used as defined in the preset. As mentioned earlier, running ctest from the command line (with --preset) works correctly.

I already tried the latest stable version (v1.18.41) and pre-release version (v1.19.3), both have the same issue.

CMake Tools Diagnostics

{
  "os": "win32",
  "vscodeVersion": "1.89.1",
  "cmtVersion": "1.19.3",
  "configurations": [
    {
      "folder": "****************",
      "cmakeVersion": "3.28.0",
      "configured": true,
      "generator": "Ninja",
      "usesPresets": true,
      "compilers": {
        "C": "cl.exe",
        "CXX": "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe"
      }
    }
  ],
  "cpptoolsIntegration": {
    "isReady": false,
    "hasCodeModel": false,
    "activeBuildType": "",
    "buildTypesSeen": [],
    "requests": [],
    "responses": [],
    "partialMatches": [],
    "targetCount": 0,
    "executablesCount": 0,
    "librariesCount": 0,
    "targets": []
  },
  "settings": [
    {
      "communicationMode": "automatic",
      "useCMakePresets": "auto",
      "configureOnOpen": false
    }
  ]
}

Debug Log

No response

Additional Information

No response

[wrightleft]

I'm having the same problem. It is making it impossible to use CMake Tools with a conan based project, because conan creates a preset file for CMake to find all its generated CMake modules.

[gcampbell-msft]

@nikolaseu @wrightleft Thanks for letting us know, could you all also test the previous official release version before 1.18.41? I'm wanting to understand if this is a regression, or if this is an already present issue. Thanks!

[gcampbell-msft]

@nikolaseu For context on why we don't use --preset, this is for numerous reasons, but the simplest being that the guidance provided by Kitware for IDE integration is to avoid using --preset. There are additional reasons based on implementation requirements, but that is the simplest one.

[nikolaseu]

@nikolaseu For context on why we don't use --preset, this is for numerous reasons, but the simplest being that the guidance provided by Kitware for IDE integration is to avoid using --preset. There are additional reasons based on implementation requirements, but that is the simplest one.

Thanks! It's always an issue in Windows, I workaround it by setting the PATH before launching vscode but I need two different envs for debug and release 😞 . QtCreator is better in that sense (without presets at least).

I was hoping CMakePresets would help, since it works from the command line. But I was not using presets before. The previous version probably has the same issue since it's not a bug if I understand you correctly.

Maybe this could be an option? It seems useful to have something that works exactly as the command line.

[v-frankwang]

@nikolaseu Thank you for reporting this issue. Regarding this issue, we have tried to reproduce the issue using a simple demo project: my_project.zip , but we do not seem to be able to reproduce the issue you mentioned. In order to investigate further, could you please provide us with a simple project that can reproduce this issue. We are looking forward to hearing from you.

ENV: VSCode：1.90.0 CMake 工具：v1.81.41 C/C++：v1.20.5

Actual results：

[nikolaseu]

Hi @v-frankwang

That project doesn't need any DLL that is not in the PATH, you don't have any runtime dependency. How are you checking/evaluating if it's using the preset correctly?

I will try to make a basic example.

[nikolaseu]

my_project_2.zip

Using the x64-release preset in both cases, running ctest fails in vscode but works on command line.

Command line usage (from a VS2022 developer command prompt)

cmake --preset x64-release
cmake --build --preset x64-release --target all
ctest --preset x64-release

[nikolaseu]

I don't have the regular "Visual Studio" IDE but I'm told this works fine there

[v-frankwang]

@gcampbell-msft I reproduced the issue based on the project provided by the user, here are the steps to reproduce it：

ENV: VS Code: 1.90.0 CMake tools: 1.18.42 C/C++:1.20.5

1. Download this file:my_project_2.zip and open it in vscode
2. Run the command:"CMake:configure"
3. Select "x64-release"
4. Click on the Testing button in the left navigation bar

Actual result： Run Test fails when you click Run Test in the Test Explorer window, but succeeds when you run it from the command line.

[nikolaseu]

@gcampbell-msft any update? is the label "more info needed" blocking it? because there's nothing to wait for. can you check again please? there's an example for reproducing it, and @v-frankwang confirmed it already.

[v-frankwang]

@nikolaseu We're very sorry we didn't have a more granular issue tag in the first place, our developers have added this to the backlog and we'll be updating the comments as soon as we can if there's any progress!

[davidhjp01]

Having same issue, vscode-cmake does not reflect updated cacheVariables values

[v-frankwang]

@davidhjp01 Thank you very much for your comment, you can verify if the issue you mentioned will reoccur after this is fixed!

[joelkeeling]

I am experiencing something similar with the "architecture" field in my presets file.

It does not work if I use the vscode extension for performing the cmake configuration step.

It does work when I configure in either of the follow ways:

• I use the --preset argument with cmake directly.
• I copy the cmake arguments that vscode is running, append my target architecture using the cmake "-A" flag, and run that with cmake directly

[gcampbell-msft]

@joelkeeling Does your issue have to do with testing as well? If not, could you create a new issue?

Either way, could you attach your simple repro steps as well?

We definitely want to prioritize CMake Presets, so we want to understand this problem fully so that we can hopefully implement a fix.

[joelkeeling]

@joelkeeling Does your issue have to do with testing as well? If not, could you create a new issue?

Either way, could you attach your simple repro steps as well?

We definitely want to prioritize CMake Presets, so we want to understand this problem fully so that we can hopefully implement a fix.

@gcampbell-msft My issue does not have to do with testing, that is my bad... When I get a moment I will try to reproduce the issue with a simpler project. If successful, I will create a new issue.

[gcampbell-msft]

@joelkeeling No worries! Thanks for the heads up, this is still definitely something we'd want to prioritize, we will keep an eye out for your issue.

[gcampbell-msft]

@nikolaseu I just downloaded your test project zip file, and I am not consistently seeing your issue. The only issue that I'm seeing is that occasionally, you have to rebuild to ensure that the project/cache is set up correctly in order to run the tests, but then I'm able to successfully run the tests from the CMake Tools extension:

[gcampbell-msft]

It seems to me that once I fix that issue where tests aren't updated after a build, this issue goes away, could you confirm? #4148

[Yingzi1234]

@nikolaseu @gcampbell-msft This issue has been fixed on latest version 1.20.50, you can get the details below. Thanks!

[gcampbell-msft]

@nikolaseu This version of the extension will be released sometime in the next week. Thanks!