workflow-core icon indicating copy to clipboard operation
workflow-core copied to clipboard

Deleting Completed / Terminated Workflows Properly

Open JordanBrobyn opened this issue 4 years ago • 2 comments

Describe the solution you'd like I am looking to remove completed and terminated workflow history from MySQL persistence.

It would be handy to either set a flag on the workflow definition to flag for deletion on termination or completion, or set a deletion policy at startup to remove all completed workflow entries from their tables [Workflow, Subscription, Event, ExecutionError etc]

Describe alternatives you've considered Currently I have attempted to add a Middleware that runs in the PostWorkflow phase.

During this phase I check the workflow.CompleteTime.HasValue and queue a background job to delete workflow logs if the workflow is completed.

Additional context It seems however that if I run hundreds of workflows at once and they start to complete together, I start to get a lot of errors in QueueConsumer. It appears it is keeping some of the completed workflows still in queue.

Error executing item b6abed40-d14d-43df-ac08-02b6d0f9fbbb - Sequence contains no elements.
System.InvalidOperationException: Sequence contains no elements.

To enqueue a background job to delete pending workflows, I am using HangFire -> BackgroundJob.Enqueue.

I have tried adding a timeout using BackgroundJob.Schedule(..., 10 minutes from now); but I still experience the error.

JordanBrobyn avatar Aug 18 '21 21:08 JordanBrobyn

Meet similar issues when enable deleteComplete feature when using Redis provider:

Error executing item 7aed8d0e-2c86-4ec7-a0dd-caaadb2d4d42 - Value cannot be null. (Parameter 'value')

DaHaiHuha avatar Mar 28 '23 07:03 DaHaiHuha