python-tutorial
python-tutorial copied to clipboard
Testsuite: catch if exception is raised in the solution function
This is something that we could not add to #160.
The issue is that, when a solution function raises any exception, the test suite takes that result as a TEST_ERROR
, while it should be considered a FAIL
.
This is different from when a test itself raises an exception. There we correctly set the outcome to TEST_ERROR
.
The idea we came up with is to parse the stack trace (extracted from the traceback with traceback.extract_tb()
), and inspect the StackSummary
object to see if it contains any "failure".
That is, if the names of the functions present in the stack trace mention a failure or an exception raised, then we know that the exception must come from the solution code, and not from the test.
We may need to revise this implementation idea if we find a better solution.