Death tests: attempt to disable core dumps in child process.

[cf-natali]

Disabling core dumps in the child process makes tests significantly faster if the process crashes while core dumps are enabled, in particular when the address space is large or dumping cores is slow because of e.g. compression or slow IO.

See this example calling ASSERT_DEATH with a raise(SIGABRT) 10 times, on my laptop with systemd-coredump enabled: Before:

$ ./googletest/googletest-death-test-test  --gtest_filter=*CoreDump*
Running main() from /home/cf/src/googletest/googletest/src/gtest_main.cc
Note: Google Test filter = *CoreDump*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from TestForDeathTest
[ RUN      ] TestForDeathTest.TestFastDeathNoCoreDump
[       OK ] TestForDeathTest.TestFastDeathNoCoreDump (1339 ms)
[----------] 1 test from TestForDeathTest (1339 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1339 ms total)
[  PASSED  ] 1 test.

After:

$ ./googletest/googletest-death-test-test  --gtest_filter=*CoreDump*
Running main() from /home/cf/src/googletest/googletest/src/gtest_main.cc
Note: Google Test filter = *CoreDump*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from TestForDeathTest
[ RUN      ] TestForDeathTest.TestFastDeathNoCoreDump
[       OK ] TestForDeathTest.TestFastDeathNoCoreDump (4 ms)
[----------] 1 test from TestForDeathTest (4 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (4 ms total)
[  PASSED  ] 1 test.

And that's with a small address space, so in practice it can yield speedups over 1000X.

Note that the code uses prctl(PR_SET_DUMPABLE, 0) over setting RLIMIT_CORE to 0 when available, because the latter does not fully prevent generating core dumps on Linux, see https://lkml.org/lkml/2011/8/25/124 for background and the below example:

$ python3 -c "import os, faulthandler; print(os.getpid()); faulthandler._sigsegv()"; sleep 3; coredumpctl -rq | head -n2
75826
Segmentation fault (core dumped)
TIME                            PID   UID   GID SIG COREFILE  EXE
Sat 2022-08-27 10:07:19 BST   75826  1000  1000  11 present   /usr/bin/python3.9
$ ulimit -c 0; python3 -c "import os, faulthandler; print(os.getpid()); faulthandler._sigsegv()"; sleep 3; coredumpctl -rq | head -n2
76425
Segmentation fault (core dumped)
TIME                            PID   UID   GID SIG COREFILE  EXE
Sat 2022-08-27 10:07:31 BST   76425  1000  1000  11 present   /usr/bin/python3.9

[apcountryman]

@cf-natali do you intend to make the changes that have been requested or should someone else pick up this work?

[cf-natali]

Hey,

I'm sorry but I don't really have time to push this. It's actually complicated by the fact that gtest has two different ways to run subprocesses - fork and fork+exec, and in the latter case it seems that contrarily to the doc PR_SET_DUMPABLE is not inherited across execve. PR_SET_DUMPABLE also has some subtle side effects like making some files under /proc/pid/ inaccessible as regular users, which could be tricky for some workloads.