[BUG] Workflow Becomes Locked After Frequent Publish [504] Timeout

[sfmskywalker]

Background

This bug occurs when testing publishing in conjunction with the "Usable as Activity" setting. It is challenging to reproduce as it requires the system to enter a specific locked state. Initial test runs did not show any issues with publishing.

Reproduction Steps

1. Create two workflows: a parent and a child (see attached workflows).
2. Publish the child workflow multiple times, alternating between the following settings:
   • Usable as Activity: On, Auto Consume: On
   • Usable as Activity: On, Auto Consume: Off
   • Usable as Activity: Off
3. After 5-6 publishes, the child workflow becomes deadlocked.
4. Subsequent publish attempts result in a timeout.

Other Observations

1. Creating and publishing a workflow with "Usable as Activity" set to Off works as expected.
2. Enabling "Usable as Activity" and publishing causes the system to hang.
3. When a workflow is locked, the frontend does not register changes (from Published to Draft).

Error Details

The following error was observed in Datadog:

System.TimeoutException: Timeout exceeded when trying to acquire the lock
   at Medallion.Threading.Internal.DistributedLockHelpers.ThrowTimeoutIfNull[T](ValueTask`1 task, String object) in /_/src/DistributedLock.Core/Internal/DistributedLockHelpers.cs:line 139
   at Medallion.Threading.Internal.Helpers.Convert[TDerived,TBase](ValueTask`1 task, ValueTaskConversion _)
   at Elsa.Workflows.Api.Endpoints.WorkflowDefinitions.Post.Post.HandleAsync(SaveWorkflowDefinitionRequest request, CancellationToken cancellationToken)
   at FastEndpoints.Endpoint`2.ExecAsync(CancellationToken ct)
   at FastEndpoints.Endpoint`2.ExecAsync(CancellationToken ct)
   at Elsa.Http.Middleware.HttpWorkflowsMiddleware.InvokeAsync(HttpContext httpContext, IServiceProvider serviceProvider)
   at Elsa.Workflows.Api.Middleware.JsonSerializationErrorHandlerMiddleware.InvokeAsync(HttpContext httpContext)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNetCore.BlockingMiddleware.Invoke(HttpContext context) in /project/tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/AspNetCore/BlockingMiddleware.cs:line 117
   at Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNetCore.BlockingMiddleware.Invoke(HttpContext context) in /project/tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/AspNetCore/BlockingMiddleware.cs:line 117
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Attachments

workflow-definitions (25) (1).zip

[sfmskywalker]

Fixed via 23d27d49d9c52d47bf5f7b36904c22f9fb413993