XCode: [SetUp|TearDown]TestSuite invoked for each test fixture in a single suite

[polishcode]

Describe the bug

When working with C++ and XCode's XCTests, contrary to the description provided in gtest sharing resources gtest does invoke both static void SetUpTestSuite() and static void TearDownTestSuite() for each and every test fixture (TEST_F).

Steps to reproduce the bug

XCode's project is setup with gtest in a following manner:

• I have a gtest-ios static library/target in the solution; it compiles gtest-all.cc
• library with test suites/cases/fixtures links gtest-ios library (+ header inclusion paths)
• each test case/fixture is discovered by XCode in runtime: by default, XCode is able to immediately show only instances of XCTest, however when running tests, it will discover and add to list all those gtest-based
• in logs and during debugging I can clearly see, that the test suite setup/tear down methods are invoked for each text fixture, which is not what I hoped for

Does the bug persist in the most recent commit?

Yes. Both 1.11 and 1.12.1 have the same problem

What operating system and version are you using?

MacOS 12.6 XCode 14; for previous versions the same problem was present.

What compiler and version are you using?

clang provided with XCode

What build system are you using?

XCode project

Additional context

Add any other context about the problem here.

[derekmauro]

Thank you for filing a bug. The steps to reproduce the bug are a good start, but we really need all of the code, preferably as a reduced test case, before we will look into a problem.

[polishcode]

Hi @derekmauro. Will try to prepare a minimal xcode project that has a repro.

[joaofl]

I believe I am having the same issue. For example with the code below:

#include <gtest/gtest.h>

class SetupTest: public ::testing::Test {
protected:
    static void SetUpTestSuite(){
        std::cout << "+++SetUpTestSuite+++" << std::endl;
        my_int = 0;
    }

    static void TearDownTestSuite() {
        std::cout << "---TearDownTestSuite---" << std::endl;
    }

    // You can define per-test set-up logic as usual.
    void SetUp() override { }

    // You can define per-test tear-down logic as usual.
    void TearDown() override { }

    static int my_int;
};

int SetupTest::my_int;

TEST_F (SetupTest, ReadSingle) {
    SetupTest::my_int++;
    EXPECT_EQ(SetupTest::my_int, 1);
}

TEST_F (SetupTest, PeersList) {
    SetupTest::my_int++;
    EXPECT_EQ(SetupTest::my_int, 2);
}

TEST_F (SetupTest, ReadAll) {
    SetupTest::my_int++;
    EXPECT_EQ(SetupTest::my_int, 3);
}

I get the following, which on my understanding is not the expected behavior:

$ ninja && ctest -R "SetupTest.*" -V
[2/2] Linking CXX executable Networking/Test/NetworkingTests
UpdateCTestConfiguration  from :/home/user/player/cmake-build-debug/DartConfiguration.tcl
Parse Config file:/home/user/player/cmake-build-debug/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/user/player/cmake-build-debug/DartConfiguration.tcl
Parse Config file:/home/user/player/cmake-build-debug/DartConfiguration.tcl
Test project /home/user/player/cmake-build-debug
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 182
    Start 182: SetupTest.ReadSingle

182: Test command: /home/user/player/cmake-build-debug/Networking/Test/NetworkingTests "--gtest_filter=SetupTest.ReadSingle" "--gtest_also_run_disabled_tests"
182: Test timeout computed to be: 1500
182: Running main() from gmock_main.cc
182: Note: Google Test filter = SetupTest.ReadSingle
182: [==========] Running 1 test from 1 test suite.
182: [----------] Global test environment set-up.
182: [----------] 1 test from SetupTest
182: +++SetUpTestSuite+++
182: [ RUN      ] SetupTest.ReadSingle
182: [       OK ] SetupTest.ReadSingle (0 ms)
182: ---TearDownTestSuite---
182: [----------] 1 test from SetupTest (0 ms total)
182:
182: [----------] Global test environment tear-down
182: [==========] 1 test from 1 test suite ran. (0 ms total)
182: [  PASSED  ] 1 test.
1/3 Test #182: SetupTest.ReadSingle .............   Passed    0.00 sec
test 183
    Start 183: SetupTest.PeersList

183: Test command: /home/user/player/cmake-build-debug/Networking/Test/NetworkingTests "--gtest_filter=SetupTest.PeersList" "--gtest_also_run_disabled_tests"
183: Test timeout computed to be: 1500
183: Running main() from gmock_main.cc
183: Note: Google Test filter = SetupTest.PeersList
183: [==========] Running 1 test from 1 test suite.
183: [----------] Global test environment set-up.
183: [----------] 1 test from SetupTest
183: +++SetUpTestSuite+++
183: [ RUN      ] SetupTest.PeersList
183: /home/user/player/Networking/Test/FailingSetUpTestSuite_test.cpp:34: Failure
183: Expected equality of these values:
183:   SetupTest::my_int
183:     Which is: 1
183:   2
183: [  FAILED  ] SetupTest.PeersList (0 ms)
183: ---TearDownTestSuite---
183: [----------] 1 test from SetupTest (0 ms total)
183:
183: [----------] Global test environment tear-down
183: [==========] 1 test from 1 test suite ran. (0 ms total)
183: [  PASSED  ] 0 tests.
183: [  FAILED  ] 1 test, listed below:
183: [  FAILED  ] SetupTest.PeersList
183:
183:  1 FAILED TEST
2/3 Test #183: SetupTest.PeersList ..............***Failed    0.00 sec
test 184
    Start 184: SetupTest.ReadAll

184: Test command: /home/user/player/cmake-build-debug/Networking/Test/NetworkingTests "--gtest_filter=SetupTest.ReadAll" "--gtest_also_run_disabled_tests"
184: Test timeout computed to be: 1500
184: Running main() from gmock_main.cc
184: Note: Google Test filter = SetupTest.ReadAll
184: [==========] Running 1 test from 1 test suite.
184: [----------] Global test environment set-up.
184: [----------] 1 test from SetupTest
184: +++SetUpTestSuite+++
184: [ RUN      ] SetupTest.ReadAll
184: /home/user/player/Networking/Test/FailingSetUpTestSuite_test.cpp:39: Failure
184: Expected equality of these values:
184:   SetupTest::my_int
184:     Which is: 1
184:   3
184: [  FAILED  ] SetupTest.ReadAll (0 ms)
184: ---TearDownTestSuite---
184: [----------] 1 test from SetupTest (0 ms total)
184:
184: [----------] Global test environment tear-down
184: [==========] 1 test from 1 test suite ran. (0 ms total)
184: [  PASSED  ] 0 tests.
184: [  FAILED  ] 1 test, listed below:
184: [  FAILED  ] SetupTest.ReadAll
184:
184:  1 FAILED TEST
3/3 Test #184: SetupTest.ReadAll ................***Failed    0.00 sec

The following tests passed:
        SetupTest.ReadSingle

33% tests passed, 2 tests failed out of 3

Total Test time (real) =   0.02 sec

The following tests FAILED:
        183 - SetupTest.PeersList (Failed)
        184 - SetupTest.ReadAll (Failed)
Errors while running CTest
Output from these tests are in: /home/user/player/cmake-build-debug/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

[derekmauro]

@joaofl - Your code is exhibiting the expected behavior.

[joaofl]

@derekmauro Indeed. It was my bad. I just realized that ctest does split each test as a separate process, reason why the fixtures do not hold between tests. Also found out that running the GoogleTest binary, the fixture works just fine. It is explained here: https://gitlab.kitware.com/cmake/cmake/-/issues/17301