Timing issues when running certain tests

[kamikazechaser]

Certain tests fail on CI when checking for time equivalence. Though they usually pass on local setups.

E.g. https://github.com/hibiken/asynq/actions/runs/8964805059/job/24617176259

1 fix for a similar failing test is available at #876