arq
arq copied to clipboard
timeout inconsitencies in case of using Retry ?
I'm trying to better understand the various timeout options and play a little with Retry,
my ultimate goal is to have a job running every x (with raise Retry(defer=x)
) that times out after y (this is where I dont get if I should use _expires
on enqueue_job or timeout
on the func or job_timeout
on the Worker, I would have bet on this one but seems like only _expires does what I want...-)
I encounter some unatural behaviours, so I'm not sure I get all the implications of the various kwargs one can set or if there are inconcistencies, or if it is a misusage on my side while trying to combine them
let's consider the following test
retry_attempts = [
# _expires, max_tries, timeout, job_timeout, defer
(None, 5, None, 300, 1),
(None, 5, None, 2, 1),
(5, 8, None, 2, 1),
(6, 5, None, 2, 1),
(None, 8, None, 2, 3),
]
retry_ids = ["defaults", "doesn't hit job_timeout 1", "hit_expires", "doesn't hit job_timeout 2", "weird"]
@pytest.mark.parametrize("_expires, max_tries, timeout, job_timeout, defer", retry_attempts, ids=retry_ids)
async def test_job_retry_with_timeout(arq_redis: ArqRedis, worker, caplog, _expires, max_tries, timeout, job_timeout, defer):
async def check_instant_job(ctx):
await asyncio.sleep(0)
return False
async def foo(ctx):
success = await check_instant_job(ctx)
if not success:
raise Retry(defer=defer)
else:
return True
caplog.set_level(logging.INFO)
print()
await arq_redis.enqueue_job('foo', _job_id='testing', _expires=_expires)
worker: Worker = worker(functions=[func(foo, name='foo', max_tries=max_tries, timeout=timeout)], job_timeout=job_timeout)
await worker.main()
-
default seems correct, it hits max_tries, all defaults timeouts are above 5*1s
-
in the 2nds parametric test, I naively thought it would hit the job_timeout after 2s, instead it goes up to max_retries which is above, in this case I retry a job
foo
every 1s, my worker has a 2sjob_timeout
, and my func a max_tries of 5 -
ok seems like _expires is doing the behaviour I want
-
same deal with test 4 as in test 2, where only the _expires is different, the job hits max_tries while it is set to timeout in 2s, so it's as if the retries didnt add up for that job_timeout
-
edit: it is the pytest-timeout plugin here