[RFC] Ability to identify jobs with unique key

[erikreedstrom]

trafficstars

Problem

We are currently using a new job state tracking mechanism based on some of the ideas provided in Introducing Centrifuge as well as Implementing Stripe-like Idempotency Keys.

Part of this implementation requires a unique identifier for work being done, however, Messages currently encode without the option for a unique identifier.

Solution

We are suggesting options be passed to the Message during encoding which can contain an id property. This would allow for the ability to identify explicit job executions without interfering with current paradigms.

Example

The following illustrates a possible solution. It is not complete, but illustrates the touch points required to support such a system.

task_bunny/lib/task_bunny/job.ex

def enqueue!(job, payload, options \\ []) do
  ...
  message_id = options[:message_id]
    {:ok, message} = Message.encode(job, payload, id: message_id)
  ...
end

task_bunny/lib/task_bunny/message.ex

def encode(job, payload, options \\ []) do
  data = message_data(job, payload, options)
  Poison.encode(data, pretty: true)
end

defp message_data(job, payload, options) do
  %{
    "job" => encode_job(job),
    "payload" => payload,
    "created_at" => DateTime.utc_now(),
    "id" => options[:id]
  }
end

[erikreedstrom]

We are forking to support this for ourselves and will submit a PR.

[erikreedstrom]

This appears more complex than we initially thought. To match what we want to accomplish, task bunny needs to expose the lifecycle of the job. To do this while remaining flexible, we’d need to introduce something like callbacks to expose the higher level meta and allow the job to determine if it should short circuit or not. Similar to #61