Unable to run a cron every 5 seconds, when job_id is set? (does the minimum have to be 60 seconds?)

[epicwhale]

I need to run a cron in my worker every 5 seconds, but I also want to provide it a job_id so there's no duplicates running in parallel. The next cron task for the job should only run if the previous one is completed.

My cron definition:

    cron(
        my_task,
        unique=True,
        job_id="cron_my_task",  # to prevent duplicate task runs
        second=set(range(0, 60, 5)),         # every 5 seconds
        keep_result=0, 
        max_tries=1,
    ),

But when I configure a worker with this, I notice there's an in-progress entry created with a TTL of 60 seconds, which prevents any future cron runs from running till it expires, even if my tasks is completed in a few seconds.

Am I missing something important here? As this behavior feels rather unpredictable, and undocumented.

Digging deeper into the code, I see there's a keep_cronjob_progress = 60 <--- # how long to keep the "in_progress" key after a cron job ends to prevent the job duplication which is a constant that can't be modified: https://github.com/samuelcolvin/arq/blob/1315583f170512fc63622af0200a2855e057a58d/arq/constants.py#L12

What's the recommended way to accomplish my requirement of running a task every 5 seconds, provided the previous cron run of that task is completed?

[eigenein]

Apparently, there is another symptom to that. Even if one schedules a run every minute, effectively the in-progress key expires 1 minute after the job finishes. So, the key would actually expire after 1 minute + <execution time>. Meaning, that every run will get more and more delayed, <execution time> longer after each run