running ctest using test explore with env vars from test presets

[mikirothman]

Brief Issue Summary

hello our test presets have bunch of environment variables (composed by conan due to our dependency graph) which we need in order to launch the tests, we were expecting to see those vars along the ctest command. but the command ctest invokes is:

ctest.exe -T test --output-on-failure --build-config Debug -R ^MyProj\.MyTest$

Due to the missing env vars the tests fail when running from the test explorer. Running ctest —preset ${my-preset} from the terminal pass the env vars to my test executables

CMake Tools Diagnostics

Debug Log

Additional Information

No response

[Yingzi1234]

@mikirothman Thank you for your github issue! To be able to reproduce your question better, can you help us answer the following question?

1. How do you run CTest? Did you run the test through VSCode's CTest Explorer or did you run the CTest command manually? If VSCode, please confirm how VSCode invokes ctest? (e.g., did it use the CTest: Run Tests button?) If running manually, please provide the full command and terminal output. 2.
2. how are your CMake presets (CMakePresets.json) defined? Could you share CMakePresets.json or the relevant CTest preset section? (especially the environment field to see if the environment variables are actually defined in the preset)

[mikirothman]

1. How do you run CTest? We tried with vscode, both with Run CTest button from the Status bar and from the test explorer. we doubled checked that the active preset is configured properly with the desired preset. when running manually, everything works as expected.

2. how are your CMake presets (CMakePresets.json) defined? Not sure I understand the question; they are defined according to docs. we have several environment variables defined in the preset, which are missing from the test executables when running from vscode. the fact that running from command line with the desired preset is kinda of a proof that the issue is not at the preset side.

[Yingzi1234]

@mikirothman @gcampbell-msft We can reproduce this issue on latest version CMake Tools1.21.2(pre-release), you can get the details below. Thank you!

Environment: Linux: Ubuntu 22.04.4LTS

Repro steps:

1. Download this folder ctest_env_test.tar.gz and open it with VSCode
2. Click F1 to run command CMake: configure
3. Open terminal window and run below code

cmake --preset default
ctest --preset default

4. Check if can be run successfully
5. Open the Testing page and click on the Run Test button of Myproj.MyTest line
6. Check if can be run successfully

Actual result: Running CTest manually works, but running it inside VSCode fails

[JonasToth]

@gcampbell-msft I did investigate the issue and patched the plugin locally to get it "running". The following findings hopefully help to work on this issue. This is right now a bigger obstacle for my organisation to adopt VSCode.

1. when ctest is executed the preset itself is not provided, but each element of the preset is passed directly to the command line. This is not necessary, as passing ["--preset", testPreset.name] suffices
   • just do return ['--preset', driver.testPreset.name]; here
   • extending the array of arguments might work too. I guess --preset has to be passed early on.
   • it is necessary to NOT pass -T test, otherwise the test discovery does not seem to work
2. ctest is executed from the binaryDir instead the sourceDir in case the preset is used directly. That leads to the issue, that the preset itself is not found. This has to be adjusted too.
   • in runCTestImpl
   • twice in refreshTests and here too
   • executing from the sourceDir creates the Testing/ directory and so on in the sourceDir. They should rather live in the matching binaryDir. It feels, that this can be fixed by passing the proper arguments, I probably botched the whole execution with my early return in 1.

With these adjustments I got the basic cases running. Changing the execution directory is of course a breaking change. This could be done conditionally or the extensive argument passing could be replace by simply using ctest --preset <test preset name>.

I hope this helps and speeds up a resolution :)

[Evelyn-001]

@JonasToth ,thanks for proving the above information. Our development team is investigating this issue. We will get back to you if there are any updates. Thanks for your support. @gcampbell-msft FYI

[hagitLeviBenHayon]

Hello, is there an estimated timeline for resolving this issue?

[yanghhhhhhh]

@hagitLeviBenHayon Thanks for the comment, we're still investigating the issue and we'll let you know as soon as there's any update. Thanks.