[BUG] Concurrency Issue in BulkDispatchWorkflows Activity Prevents Completion

[sfmskywalker]

Summary

The BulkDispatchWorkflows activity fails to complete due to a concurrency issue introduced during an update. This issue arises when updating the workflow instance to store workflow input before sending a message to the service bus.

Problem

When we implemented support for storing workflow input with the workflow instance to prevent large message sizes, we inadvertently created a situation where multiple threads could attempt to update the workflow instance state simultaneously. This lack of thread synchronization has led to the BulkDispatchWorkflows activity failing to complete.

Solution

To resolve this issue, we need to acquire a distributed lock on the workflow instance before updating it with the input. This will ensure that only one thread can update the workflow instance at a time, preventing concurrency conflicts.

[sfmskywalker]

This issue still exists.

[sfmskywalker]

Fixed