exq

exq copied to clipboard

This PR implement 2 timeout options for exq.

• queuing_timeout (in second): TTL for job in queue, timed out job will be discard using a middle ware
• execution_timeout (in second): if execution time is longer than execution_timeout the worker is killed and not retry any more.

This could be used as an example for those who want to implement timeout. This PR does not included test and execution

Both of these can be implemented easily with existing APIs/middleware provided by Exq. I am not convinced about the benefit of adding this in Exq itself.

expiration

def perform() do
  job = Exq.worker_job()
  if !expired?(job.enqueued_at) do
    do_work()
  end
end

timeout

def perform() do
  {:ok, tref} = :timer.kill_after(:timer.seconds(timeout))

  try do
    do_work()
  after
    :timer.cancel(tref)
  end
end

Disable retry

for certain errors like the kill above

defmodule RetryMiddleware do
  def before_work(pipeline), do: pipeline
  def after_processed_work(pipeline), do: pipeline

  def after_failed_work(pipeline) do
    if non_retryable?(pipeline.assigns.error) do
      put_in(pipeline.assigns.job.retry, pipeline.assigns.job.retry_count || 0)
    else
      pipeline
    end
  end
end

@ananthakumaran wow thank you for your simple solution