fprime
fprime copied to clipboard
Undef sanitizer kills unit tests
The undefined behavior sanitizer kills tests that use undefined behavior to drive off-nominal cases. For example, synthesizing an invalid On/Off command argument with the value 2 causes the test to be aborted in the harness, before the command handler is run. There are a couple of issues here:
- The test is aborted with a runtime error, but GTest still reports success. It seems that runtime errors should cause test failures.
- The test abort causes tests not to be run. For example, the “invalid command argument” test becomes a no-op that doesn’t exercise any code and always passes.
We should ensure that sanitizers tripping also fail the unit test, otherwise the use of these sanitizers is meaningless as no one will see or respond to the results.
I am not convinced on the use of undefined behavior to drive a unit test. Undefined behavior is not guaranteed to work in a reliable way and its usage, even in a test, would mean the code cannot be relied upon. For example, a test's pass/failure state might suddenly depend on the compiler, optimization level, or some other factor.