Skipping a signal test is marked as an error

[COLAMAroro]

Criterion version: 2.3.3 OS: Fedora 30 Compiler: GCC 9.2.1

#include <criterion/criterion.h>
#include <signal.h>
#include <string.h>

static void skip(void)
{
    cr_skip_test("Minimal skip");
}

TestSuite(c, .init=skip);

Test(c, should_skip, .signal=SIGSEGV)
{
    strlen(NULL);
}

Test(c, should_skip_too)
{
    cr_assert(1);
}

Excepted behaviour: Either both test skipping (if a skip is allowed in a TestSuite init), or both should be valid (if it's not allowed) Actual behaviour:

[0][cola@FedoKeon]%->gcc minimal_error.c -lcriterion                                                             
[0][cola@FedoKeon]%->./a.out --verbose                                                                           
[----] Criterion v2.3.3
[====] Running 2 tests from c:
[RUN ] c::should_skip
[FAIL] c::should_skip: (0,00s)
[RUN ] c::should_skip_too
[SKIP] c::should_skip_too: Minimal skip
[====] Synthesis: Tested: 1 | Passing: 0 | Failing: 1 | Crashing: 0

[Snaipe]

Yeah, that happens due to the way that exit status get validated: skipping a test makes the test exit with status 0, while c::should_skip expects to die with SIGSEGV.

I'd need to add an edge-case to ignore the exit status if the test was skipped.

[COLAMAroro]

I tried fixing it on my own (but sadly failed), but why not exit with attended value ? If a test want the exit code to be 3, a skip would exit with code 3

[Snaipe]

why not exit with attended value ? If a test want the exit code to be 3, a skip would exit with code 3

It's dicier than that because the skip would need to also kill() the current process with the expected signal, and even then it wouldn't be able to lie to the parent on the circumstances where this happened. Overall, I think it's bad form over just handling the corner case correctly.

I'll look at whether this can be easily fixed, but immersing myself in that part of the code has been fairly challenging over the last couple of days.