touchstone icon indicating copy to clipboard operation
touchstone copied to clipboard

Assertions Automatic Retry w/Timeout (alternative to processing_period sleep)

Open ScottFreeCode opened this issue 3 years ago • 0 comments

Right now, in order to give tests enough time for Rabbit messages to be processed, a hardcoded delay is introduced. Set too low and the tests become flakey, set higher and they are more reliable but take longer. It would be best if instead there was a timeout, either at the test level or the assertion level, for trying the assertions until they either pass or have continued failing till the timeout is hit.

Since Touchstone currently prints the test failure from within the assertion on failure, some refactoring will be needed to prevent printing assertion failures over and over.

I can think of a couple different possible solutions. They aren't mutually exclusive and could be used together to get the advantages of both.

Solution A: Throw

Most test frameworks I'm familiar with throw an exception on failed assertions, where the exception message describes the failure and the assertion exception class contains an expected value and an actual value.

This leaves it up to the test framework (or a reporting plugin for the framework; see #15) how to print the message and the comparison of expected vs actual.

Solution B: Retry Within Assertion

Cypress assertions do not actually fail until they've tried until a time limit is hit. Touchstone assertions could do likewise. There could be a default time limit for the test with a possibility to override it for individual assertions.

ScottFreeCode avatar Dec 03 '20 21:12 ScottFreeCode