Workless gem does not distinguish between future scheduled jobs and immediate execution jobs

[kynatro]

We recently encountered a scenario where we started taking advantage of the scheduling capability of Delayed Job in our application and encountered a serious and costly problem used in conjunction with Workless. Workless doesn't seem to be able to distinguish between jobs that are set to execute as soon as possible in the queue and jobs that are scheduled for a far future date. As a result, the 1,100 or so future scheduled events are sitting in the queue, not being executed on, but being seen as a large queue backlog nonetheless. Workless sees this large queue backlog and is automatically scaling things far beyond what is necessary for jobs that aren't even being processed at that time, resulting in multiple worker servers sitting active, but completely idle, costing us money for every hour they're up.

There should probably be a minimum threshold for the run_at column to be at before the job is considered in the queue to be processed. Maybe an option for scaling based off backlogs in specific queues or priorities could also allow this issue to be worked around.

[davidakachaos]

workless currently doesn't support run_at with a date in the future. It will do what you describe, which is of course not what we want. There would be an issue for workless missing a job in the future when we don't spin up a worker at all (have 0) I think we could make a change so we only count the number of pending jobs with a run_at that is now or has past. We then would only need 1 worker to be running all the time. If we don't have a worker running, the scheduled jobs wouldn't be run because workless only scales up or down when a new job is inserted... But we can get around that by always running 1 worker... What do you think? Does that sound a bit sensible?

[kynatro]

This sounds good to me. You may want to make support for future jobs something that could be toggled since the current operating paradigm is that an application could work with 0 dynos spooled. That being said, this will be perfectly fine for my application as I always have at least one worker dyno spooled up.