arq icon indicating copy to clipboard operation
arq copied to clipboard

timeout inconsitencies in case of using Retry ?

Open euri10 opened this issue 2 years ago • 0 comments

I'm trying to better understand the various timeout options and play a little with Retry,

my ultimate goal is to have a job running every x (with raise Retry(defer=x)) that times out after y (this is where I dont get if I should use _expires on enqueue_job or timeout on the func or job_timeout on the Worker, I would have bet on this one but seems like only _expires does what I want...-)

I encounter some unatural behaviours, so I'm not sure I get all the implications of the various kwargs one can set or if there are inconcistencies, or if it is a misusage on my side while trying to combine them

let's consider the following test

retry_attempts = [
	# _expires, max_tries, timeout, job_timeout, defer
    (None, 5, None, 300, 1),
    (None, 5, None, 2, 1),
    (5, 8, None, 2, 1),
    (6, 5, None, 2, 1),
    (None, 8, None, 2, 3),
]
retry_ids = ["defaults", "doesn't hit job_timeout 1", "hit_expires", "doesn't hit job_timeout 2", "weird"]


@pytest.mark.parametrize("_expires, max_tries, timeout, job_timeout, defer", retry_attempts, ids=retry_ids)
async def test_job_retry_with_timeout(arq_redis: ArqRedis, worker, caplog, _expires, max_tries, timeout, job_timeout, defer):

    async def check_instant_job(ctx):
        await asyncio.sleep(0)
        return False

    async def foo(ctx):
        success = await check_instant_job(ctx)
        if not success:
            raise Retry(defer=defer)
        else:
            return True
    caplog.set_level(logging.INFO)
    print()
    await arq_redis.enqueue_job('foo', _job_id='testing', _expires=_expires)
    worker: Worker = worker(functions=[func(foo, name='foo', max_tries=max_tries, timeout=timeout)], job_timeout=job_timeout)
    await worker.main()
  1. default seems correct, it hits max_tries, all defaults timeouts are above 5*1s

  2. in the 2nds parametric test, I naively thought it would hit the job_timeout after 2s, instead it goes up to max_retries which is above, in this case I retry a job foo every 1s, my worker has a 2s job_timeout, and my func a max_tries of 5

  3. ok seems like _expires is doing the behaviour I want

  4. same deal with test 4 as in test 2, where only the _expires is different, the job hits max_tries while it is set to timeout in 2s, so it's as if the retries didnt add up for that job_timeout

  5. edit: it is the pytest-timeout plugin here

euri10 avatar Nov 14 '22 16:11 euri10