fprime icon indicating copy to clipboard operation
fprime copied to clipboard

Undef sanitizer kills unit tests

Open bocchino opened this issue 1 year ago • 1 comments

The undefined behavior sanitizer kills tests that use undefined behavior to drive off-nominal cases. For example, synthesizing an invalid On/Off command argument with the value 2 causes the test to be aborted in the harness, before the command handler is run. There are a couple of issues here:

  1. The test is aborted with a runtime error, but GTest still reports success. It seems that runtime errors should cause test failures.
  2. The test abort causes tests not to be run. For example, the “invalid command argument” test becomes a no-op that doesn’t exercise any code and always passes.

bocchino avatar Oct 18 '23 20:10 bocchino

We should ensure that sanitizers tripping also fail the unit test, otherwise the use of these sanitizers is meaningless as no one will see or respond to the results.

I am not convinced on the use of undefined behavior to drive a unit test. Undefined behavior is not guaranteed to work in a reliable way and its usage, even in a test, would mean the code cannot be relied upon. For example, a test's pass/failure state might suddenly depend on the compiler, optimization level, or some other factor.

LeStarch avatar Oct 19 '23 17:10 LeStarch