Check for ASSERTs not executed

[agroce]

Without compiling with coverage, we could have DeepState warn you at finish of internal fuzzer, test directory replay, or symex, if you had ASSERTs in your DeepState harness that never executed. Seems like a nice cheap sanity check.

https://hal.inria.fr/hal-02002346/document

[pgoodman]

So how would this work in practice? It seems like we'd need some call graph analysis. We can get that from Angr's CFGFast, most likely. If libFuzzer is used then maybe we could also inject our own LLVM instrumentation that records info. Another alternative could be some epic hacks...

Layout out an incomplete solution:

Each ASSERT macro could inject some inline assemble that, in a different section, puts the location of the assert and function name and file name, some address back into the function, and maybe the address of the hit count.

From here we could parse out a grouping of (file, function) -> [asserts]. We could also get a kind of min_pc and max_pc for the asserts in the function. So, if we hit any assert in the range of (min_pc <= pc <= max_pc), then we can figure out the total number of asserts we expect to hit in that function.

So far, this doesn't get us beyond a single function though, that is, if a test doesn't set x correctly, then we wouldn't know about the unhit ASSERT in foo. This may be fine...

void foo() {
  ASSERT(..);
}
void bar(int x) {
  if (x) {
    foo();
  }
  ASSERT(...);
}

One thing we could maybe do is ask people to compile their code with gprof instrumentation, which we implement in libdeepstate.a. Then see what we could do from there.

[agroce]

Ok, yeah, maybe cheap but a pain to implement.

[agroce]

Ok, dirt cheap and easy via macros, but less user friendly:

each ASSERT marks itself (file, line no) as hit in a global structture. Then we tell users which ASSERTs were hit, not which were missed.

[agroce]

E.g., output at end is something like:

ASSERT foo.c:45   executed 14567 times
ASSERT foo.c:165  executed 205 times

[pgoodman]

Well I think we can say partially which asserts were missed. That is, if you've hit one assert in a function, and there's a second not hit one, then I think we could say that you didn't hit it. But anything one or two levels removed won't be recognized.