Disabled gtest tests are executed and fail

[rokas-kbit]

Hi,

It seems that extension is handling disabled gtest tests incorrectly.

Current behavior

Disabled tests are executed when execution of all tests is requested and such tests are considered as failed:

[proc] Executing command: /usr/local/bin/ctest -j18 -C Debug -T test --output-on-failure -R ^Tests\.Runs$
[ctest]    Site: rk-work-laptop
[ctest]    Build name: Linux-c++
[ctest] Test project /home/rokas/cmake-tools-bug/build
[ctest]     Start 1: Tests.Runs
[ctest] 1/1 Test #1: Tests.Runs .......................   Passed    0.00 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.00 sec
[ctest] CTest finished with return code 0
[proc] Executing command: /usr/local/bin/ctest -j18 -C Debug -T test --output-on-failure -R ^Tests\.DoesntRun$
[ctest]    Site: rk-work-laptop
[ctest]    Build name: Linux-c++
[ctest] Test project /home/rokas/cmake-tools-bug/build
[ctest]     Start 2: Tests.DoesntRun
[ctest] 1/1 Test #2: Tests.DoesntRun ..................***Not Run (Disabled)   0.00 sec
[ctest] No tests were found!!!
[ctest] CTest finished with return code 0
[ctest] Test Tests.DoesntRun failed with completion status "Disabled".

Expected behavior

• Disabled tests should not be executed when we are running all tests.
• They may be included in the test list and may be executed manually.
• They should not impact status of the test group.
  • Manual execution may impact group status, but it starts to get complicated. For example, if entire group of tests is executed again, we probably want disabled tests to stop impacting group status, even if we manually executed these tests and they failed.

Minimal reproducer: bug-4267.zip

[Amy-Li03]

Hi @rokas-kbit , thanks for reporting issue here! We can repro this issue on cmake tools latest build 1.20.47(pre-release), following is the repro steps:

1. Extract and open project: ctest-project.zip with VS Code
2. Build this project
3. Switch to "Testing" tab and click the "Run tests" button

Actual result: Disabled gtest tests are executed and fail

@gcampbell-msft FYI

[amir-s2]

Any update?

[yanghhhhhhh]

Hi @amir-s2, thanks for the comment. If there is progress on this issue, we will update here. Thanks for help us build a better CMake Tools!

[anand-hariharan]

I have tested and confirm that changing this line of code to -

if (completionStatus === "SKIP_REGULAR_EXPRESSION_MATCHED" || completionStatus === "Disabled") {

fixes the problem.

[yanghhhhhhh]

@anand-hariharan Thank you for your information. If this issue is fixed, we will update here. Thanks for help us build a better CMake Tools!