action-scheduler icon indicating copy to clipboard operation
action-scheduler copied to clipboard

Improve tests involving timestamp comparison (and reduce sporadic CI failures)

Open barryhughes opened this issue 4 years ago • 3 comments
trafficstars

We have various tests that check if two timestamps match. Example here. However, especially in Travis CI, there is potential for a discrepancy to arise (which usually doesn't exceed 1s).

This is a problem in practice because, it feels like, almost every 2nd PR has a bunch of CI failures which are sporadic in nature and clear up on re-running the checks (but may re-appear in different places).

...............................................................  63 / 366 ( 17%)
..........F.................................................... 126 / 366 ( 34%)
............................................................... 189 / 366 ( 51%)
............................................................... 252 / 366 ( 68%)
............................................................... 315 / 366 ( 86%)
...................................................             366 / 366 (100%)

Time: 10.16 seconds, Memory: 46.25MB

There was 1 failure:

1) ActionScheduler_DBStore_Test::test_get_run_date
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'1623948928'
+'1623948927'

:point_up: The sort of output we see. This could show up for 'PHP 7.2 WP 5.1 Multisite' in one run and 'PHP 5.4 WP 5.3 Single Site' the next—it's surfaces unpredictably and so devalues our CI checks.

We should come up with a custom assertion (ie $this->assertTimestampsAreApproximatelyEqual(...)), or some similar solution, to better accommodate this and give us a ±1s tolerance that we can apply where relevant.

barryhughes avatar Jun 17 '21 17:06 barryhughes

:memo: PR #727 doesn't actually solve this, but it certainly seems to reduce instances of the problem (and so it may be all we actually need).

barryhughes avatar Jul 08 '21 15:07 barryhughes

Setting to low priority...the potential for this still exists, but we encounter it far less now (it was especially prone to surfacing under Travis, because of that platform's performance characteristics...but of course we've now moved from that to GitHub Actions).

barryhughes avatar Sep 03 '21 18:09 barryhughes

See also #512.

barryhughes avatar Dec 21 '22 22:12 barryhughes