[TRI-1132] Add a deduplicationKey to Tasks

[matt-aitken]

What is it?

Sometimes you only want a Task to run once across your entire Project/Job.

One example is sending non-recurring emails. This should only ever happen exactly once.

await io.resend.sendEmail("welcome-email", {
  to: user.email,
  subject: "Welcome to Trigger.dev!",
  react: <WelcomeEmail user={user} />,
  from: "Trigger.dev <[email protected]>",
}, {
  deduplicationKey: `welcome-email-${user.id}`,
  //optional and defaults to "project", but can be "job" or "organization" as well
  deduplicationScope: "project"
});

It would be impossible to have another Task run with the same deduplicationKey inside a project in this case.

If you specified the scope to be "job" or "organization" it would deduplicate at that scope instead.

If it has already run a Task with the same dedupe key it would return the data, just like the key does.

Implementation

1. Add a deduplicationKey column to the Task table which is a unique String, required, and defaults to a cuid. Migration needs to add unique ids to all existing Tasks.
2. Add SDK support to runTask
3. On the server it would prepend the proj:${projectId}, org:${orgId}, job:${jobId} to the key before it's saved to the database. Also on lookup it needs to do the same.
4. Roll out support to integrations

_{From SyncLinear.com | TRI-1132}

[lazyfuhrer]

Can I go ahead @matt-aitken ?

[ericallam]

@lazyfuhrer you can go ahead and try 👍

[hmacr]

The current behaviour is to fail a job when a task ends in failure (after all retry attempts). For project & org scoped tasks, this may not be an acceptable behaviour since it'll lead to cancelling all the job runs which consist of this task. We have a couple of options under consideration.

1. Re-execute the task if it has failed before.
2. Skip re-execution of the task, but do not let it affect the current job run. Essentially, re-use the failure output of the task and continue with the rest of the job execution.

I think option 1 would be appropriate as it satisfies the intended requirement that the side effect has to be observed just once.

@matt-aitken @ericallam What would be the right way to move forward here?

[hmacr]

Or a third option would be to allow the user to pass a task option skipFailure: true | false (will be part of the RunTaskOptionsSchema). Use this option to determine if the run should proceed or fail.

I think, option 1 or 3 is the right thing to do depending on if we want to give the user control over task failures. @matt-aitken @ericallam Your thoughts?