Catch2 Implement process isolation support for tests

Major features that could be implemented afterwards:

Catch should not go crazy if the test case uses fork or similar system call
Catch should allow running tests in isolated processes
Catch should allow testing for abort -- there should be a macro along the lines of REQUIRE_ABORT(expr), that launches separate process to check expr and succeeds if the launched process aborts
Catch should allow limited time for test suite execution

Mar 11 '17 11:03 horenmar

Really, really needed feature!

Mar 01 '18 21:03 pavelponomarev

:+1:

Mar 02 '18 12:03 Leandros

@philsquared Is this still under consideration? Would really like to see this feature! :-)

Oct 30 '19 21:10 ranty-fugue

#pragma once

#include <functional>
#include <iostream>
#include <sstream>
#include <thread>
#include <unistd.h>
#include <sys/wait.h>
#include <fcntl.h>

struct catchpp_stdstream { int fd[3], target; std::stringstream ss; };

static inline bool
catchpp_fork_and_run(std::function<void(void)> fun) {
    struct catchpp_stdstream stream[] = {
        { { -1, -1, -1 }, STDOUT_FILENO, std::stringstream() },
        { { -1, -1, -1 }, STDERR_FILENO, std::stringstream() },
    };

    for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
        pipe(stream[i].fd);
        fcntl(stream[i].fd[0], F_SETFL, O_NONBLOCK);
        fcntl(stream[i].fd[1], F_SETFL, O_NONBLOCK);
    }

    pid_t pid;
    if ((pid = fork()) == 0) {
        for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
            dup2(stream[i].fd[1], stream[i].target);
            close(stream[i].fd[1]);
            close(stream[i].fd[0]);
        }
        fun();
        _exit(0);
    }

    int code;
    waitpid(pid, &code, 0);

    char buf[1024];
    for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
        close(stream[i].fd[1]);
        for (ssize_t r = 0; (r = read(stream[i].fd[0], buf, sizeof(buf))) > 0;) stream[i].ss.write(buf, r);
        close(stream[i].fd[0]);
    }

    std::cout << stream[0].ss.str();
    std::cerr << stream[1].ss.str();
    return WIFEXITED(code);
}

#define REQUIRE_ABORT(x) REQUIRE_FALSE(catchpp_fork_and_run(x))

static inline void
catchpp_catch_stdstreams(std::function<void(void)> fun, std::function<void(const std::stringstream &sout, const std::stringstream &serr)> then) {
    struct catchpp_stdstream stream[] = {
        { { -1, -1, -1 }, STDOUT_FILENO, std::stringstream() },
        { { -1, -1, -1 }, STDERR_FILENO, std::stringstream() },
    };

    for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
        pipe(stream[i].fd);
        fcntl(stream[i].fd[0], F_SETFL, O_NONBLOCK);
        fcntl(stream[i].fd[1], F_SETFL, O_NONBLOCK);
        stream[i].fd[2] = dup(stream[i].target);
        dup2(stream[i].fd[1], stream[i].target);
    }

    fun();

    char buf[1024];
    for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
        fsync(stream[i].target);
        for (ssize_t r = 0; (r = read(stream[i].fd[0], buf, sizeof(buf))) > 0;) stream[i].ss.write(buf, r);
        close(stream[i].fd[0]);
        close(stream[i].fd[1]);
        dup2(stream[i].fd[2], stream[i].target);
        close(stream[i].fd[2]);
    }

    std::cout << stream[0].ss.str();
    std::cerr << stream[1].ss.str();
    then(stream[0].ss, stream[1].ss);
}

#define CATCH_STDSTREAM(x, y) catchpp_catch_stdstreams(x, y)

Something I've used. Fork is not really safe for various things though. Isolating tests to own processes would be the right way.

Mar 22 '20 18:03 Cloudef

microsoft/GSL#831

Apr 29 '20 01:04 robert-andrzejuk

I think I've mentioned this in person but ASSERT_DEATH is the only thing keeping using GTest both at work and at home. It's vital to testing contract tests.

May 17 '20 16:05 johnmcfarlane

I think I've mentioned this in person but ASSERT_DEATH is the only think keeping using GTest both at work and at home. It's vital to testing contract tests.

+1, we might have to swap over to gtest..

Oct 06 '20 00:10 rfbird

+1 from my side too

Nov 25 '20 11:11 escherstair

Here is an idea for a design:

I imagine that we extend the existing runner Catch::Session::run() with an undocumented command-line switch --managed where you can pass two file descriptors used for communication. Then we create an external runner called catch2-runner that you run e.g like this: catch2-runner test-executable . The catch2-runner will then create two pipes and launch test-executable with test-executable --managed fd1 fd2. It will then be able to send instructions to the test-executable via one of the fds and receive responses on the other. I imagine we make a simple protocol for communication - perhaps json based. I imagine the protocol will provide catch2-runner with commands like these:

Request list all available tests with tags
Run test

The test-executable running Catch::session::run() in managed mode will be able to respond with

List of all available tests with tags
Test competion with metadata - duration and test result

catch2-runner will execute the first command to get a list of all tests and after that it will issue Run test commands every time it receives a response back about a test completing.

This would give us the following benefits

A crashing test will still give a full (junit) test report xml - only the failing test(s) will be flagged as crashing instead of what happens now which is leave an empty or missing xml report
We can reliably capture stderr and stdout (I’d like them to be joined) and include that in the junit xml report for much improved usability in CI. stderr+stdout can be thrown away for all tests that don’t fail to avoid generating huge xml files
We have a path for running multiple instances of test-executable in parallel. For large suites of tests this would be a significant win. All it would take is for catch2-runner to be able to launch multiple copies of test-executable and run different subset of tests in each.
Ability to timeout individual tests.

Doing this would require some platform specific code and/or some dependencies, ie for creating pipes, starting and managing processes and perhaps a JSON library for the wire protocol.

Mar 13 '21 11:03 tsondergaard

I have some code that calls std::terminate() if a requested/required allocation would be too large (I have exceptions disabled so throwing an exception is not an option). Would love this feature!

Apr 01 '21 03:04 Raekye

Any word on when this is being implemented? I have an assert in my code to make sure passed values match as an optimization to avoid a strlen call. Notifies others of proper usage, and gets optimized out on release builds. An exception is not exactly ideal here.