reassure
reassure copied to clipboard
callstack
Process exit code on test failure
Describe the improvement
It seems like the reassure-cli process always returns a success exit code no matter what is the result of the tests. Currently this makes it so the task that runs on CI always succeeds even if some code breaks the tests.
I think test failures usually mean something is wrong with the test vs a perf regression (unless maybe the test times out, although this could also be because of invalid scenario).
Scope of improvement
Return the error code returned by the jest process.
Suggested implementation steps
We could add a flag that drives whether tests exit non-zero on failure or not. Would that work for you?
Our assumption when running perf tests is that all of the test will work without throwing error. But I guess it make sense to actually make the reassure cli fails in case that there are some test errors during test runs. Since perf tests should not contain assertions, then the errors would come from the actual component rendering.
@thymikee do you see any cons of introducing such behavior?
@janic-cb could you create a PR for that change? It should be quite simple.
The way I imagine it would be to connect it with some kind of a threshold config option, that would drive the decision whether to fail the whole test run or not
Oh, I think I misunderstood the initial request. If an underlying test fails due to user errors, then let's just fail, same as a js test suite would. The misunderstanding on my part was that I thought you're after failing a test when it has meaningful changes to the render duration. Which is a separate feature request, but something I'd happily accept :D