Question about event deduplication

[1am]

Hello,

Is there any way to deduplicate events being pushed to the queue? I've found some dedup ocurrences but I'm not yet clear if this can be used from the client side? I have a protection mechanism for double task execution for same inputs but am looking if it could be handled on the Hatchet level when adding items.

Best regards

[abelanger5]

Hi @1am, this can be set from the client side by added a key called dedupe to the additional metadata of the workflow, and then setting the value to the field you'd like to dedupe on.

We're working on exposing this as a better deduplication API (including a count for the # of occurrences) using expression-based syntax parsing for deduplication keys. Will share an update here when we tackle that.

[1am]

Thank you @abelanger5 This sounds great but I can't seem to find how to specify dedup in metadata of the workflow when using RegisterWorkflow. I've started with similar approach as in in examples/simple/main.go Could you point me to how to call createNewWorkflowRun which seems to deal with dedupe key?

[AndryHTC]

Any news about this?

[kadimon]

dedupe key stopped working in recent releases. The cancelled DUPLICATE_WORKFLOW_RUN tasks always remain in the Running status and block the worker.

[AndryHTC]

@amoonpod did you check this?

[inev]

Is this feature broken? Tried using the examples by setting the dedupe key and triggering multiple runs and it seems to queue and run multiple times even with the same dedupe key.

EDIT: Nevermind, read that its not supported anymore. Would have been nice to have this :/

[github-actions[bot]]

This issue has been stale for 30 days. Please update the issue or comment to keep it active. Otherwise, it will be closed in 5 days.