Extract the failure message from EXPECT_*

[mark-poscablo]

I have a similar situation to this thread in google groups, where I found it necessary to use Death Tests to spawn a new process for a test (see https://github.com/arrayfire/arrayfire/pull/2514). In essence, I have a test that looks like this:

TEST(TestSuite, Test) {
    EXPECT_EXIT({
            int foo = SomeFunction();
            EXPECT_EQ(42, foo);
            if (HasFailure()) {
                fprintf(stderr, "Test failed");
                exit(1);
            } else {
                fprintf(stderr, "Test succeeded");
                exit(0);
            }
        }, ::testing::ExitedWithCode(0), "Test succeeded");
}

Now when that EXPECT_EQ call produces a non-fatal failure, it outputs something like this:

[ RUN      ] TestSuite.Test
/foo/bar/test.cpp:475: Failure
Death test: { int foo = SomeFunction(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal:: EqHelper<(sizeof(::testing::internal::IsNullLiteralHelper(42)) == 1)>::Compare("42", "foo", 42, foo))) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalFailure, "/Users/mark-poscablo/Documents/ArrayFire/arrayfire/test/backend.cpp", 466, gtest_ar.failure_message()) = ::testing::Message(); if (HasFailure()) { fprintf(__stderrp, "Test failed"); exit(1); } else { fprintf(__stderrp, "Test succeeded"); exit(0); } }
    Result: died but not with expected exit code:
            Exited with exit status 1
Actual msg:
[  DEATH   ] Test failed
[  FAILED  ] TestSuite.Test (21 ms)

Which doesn't help much, especially for a real test, because it doesn't tell me what's the actual failed assertion. Really, what I want to see is something like this:

[ RUN      ] TestSuite.Test
/foo/bar/test.cpp:475: Failure
Death test: { int foo = SomeFunction(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal:: EqHelper<(sizeof(::testing::internal::IsNullLiteralHelper(42)) == 1)>::Compare("42", "foo", 42, foo))) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalFailure, "/Users/mark-poscablo/Documents/ArrayFire/arrayfire/test/backend.cpp", 466, gtest_ar.failure_message()) = ::testing::Message(); if (HasFailure()) { fprintf(__stderrp, "Test failed"); exit(1); } else { fprintf(__stderrp, "Test succeeded"); exit(0); } }
    Result: died but not with expected exit code:
            Exited with exit status 1
Actual msg:
[  DEATH   ] Expected equality of these values:
[  DEATH   ]   42
[  DEATH   ]   foo
[  DEATH   ]     Which is: 0
[  DEATH   ] Test failed
[  FAILED  ] TestSuite.Test (21 ms)

And to do that, I figured that if only I can extract the string from the stream that EXPECT_EQ uses and print that to stderr, I'll be able to print out something like the above. I've looked in the googletest source for anything that could help, and I did see AssertionResult::message(), so I'm looking for a way to get that from the EXPECT_* calls without implementing my own assert function that returns an AssertionResult. Is there any way within the public API to do that?

[tkocmathla]

I have the same need, and after mucking around with gtest internals for a few days, I found that test events are explicitly suppressed in the child process of a death test (at least for --gtest_death_test_style=fast). I commented out the call to SuppressEventForwarding() here, and was able to see the "desired" version of the output you shared. Unfortunately, that function and its inverse, EventForwardingEnabled(), are private.

There's probably a good reason for the suppression, but maybe a better solution would be for gtest to redirect the events to a temporary listener that can be queried after a death test runs.

[tkocmathla]

good news, it looks like 18fa6a4 addresses this issue! i pulled it locally and can confirm it does what i want.

[lipeng28]

Hi @tkocmathla I tried the GTest 1.14, however, the test events are still suppressed in the child process of a death test. Can I ask which version you used to verify the fix?

good news, it looks like 18fa6a4 addresses this issue! i pulled it locally and can confirm it does what i want.

[lipeng28]

Another question, do you think suppression for test events in the child process of a death test is the default behavior or a bug? Thanks

[tkocmathla]

Hi @tkocmathla I tried the GTest 1.14, however, the test events are still suppressed in the child process of a death test. Can I ask which version you used to verify the fix?

good news, it looks like 18fa6a4 addresses this issue! i pulled it locally and can confirm it does what i want.

as i said, i used 18fa6a4 (there was no release at the time that included this commit)

[tkocmathla]

Another question, do you think suppression for test events in the child process of a death test is the default behavior or a bug? Thanks

it's the intended behavior (see my first message that has a link to the source)

[lipeng28]

Hi @tkocmathla I tried the GTest 1.14, however, the test events are still suppressed in the child process of a death test. Can I ask which version you used to verify the fix?

good news, it looks like 18fa6a4 addresses this issue! i pulled it locally and can confirm it does what i want.

as i said, i used 18fa6a4 (there was no release at the time that included this commit)

I tried the commit 18fa6a4, the output still got suppressed, did you turn on any options? I think V1.14 contains this commit, right?

#include <gtest/gtest.h>

using ::testing::InitGoogleTest;

int SomeFunction() {
  return 7;
}

bool HasFailure() {
  return true;
}

TEST(TestSuite, Test) {
  EXPECT_EXIT(
      {
        int foo = SomeFunction();
        EXPECT_EQ(42, foo);
        if (HasFailure()) {
          fprintf(stderr, "Test failed");
          exit(1);
        } else {
          fprintf(stderr, "Test succeeded");
          exit(0);
        }
      },
      ::testing::ExitedWithCode(0),
      "Test succeeded");
}

int main(int argc, char** argv) {
  InitGoogleTest(&argc, argv);
  int ret_val = RUN_ALL_TESTS();
  return ret_val;
}

and the output is:

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from TestSuite
[ RUN      ] TestSuite.Test
main.cc:14: Failure
Death test: { int foo = SomeFunction(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("42", "foo", 42, foo))) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalFailure, "main.cc", 17, gtest_ar.failure_message()) = ::testing::Message(); if (HasFailure()) { fprintf(stderr, "Test failed"); exit(1); } else { fprintf(stderr, "Test succeeded"); exit(0); } }
    Result: died but not with expected exit code:
            Exited with exit status 1
Actual msg:
[  DEATH   ] Test failed

[  FAILED  ] TestSuite.Test (0 ms)
[----------] 1 test from TestSuite (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (0 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] TestSuite.Test

 1 FAILED TEST

[tkocmathla]

Hi @tkocmathla I tried the GTest 1.14, however, the test events are still suppressed in the child process of a death test. Can I ask which version you used to verify the fix?

good news, it looks like 18fa6a4 addresses this issue! i pulled it locally and can confirm it does what i want.

as i said, i used 18fa6a4 (there was no release at the time that included this commit)

I tried the commit 18fa6a4, the output still got suppressed, did you turn on any options? I think V1.14 contains this commit, right?

oh i see the problem. what that commit actually did was to make the SuppressEventForwarding function public. so you just need to call it inside your EXPECT_EXIT function:

::testing::UnitTest::GetInstance()->listeners().SuppressEventForwarding(false);