Concurrent automation processing

[Rory-Powell]

Describe the feature request Currently automation can process at a max rate limit of 1 automation per app-service instance, i.e. In a clustered environment with 10 app-services this results in 10 concurrent automations at any one time. This limit can be significantly increased by enabling concurrency on bull queue processing.

There are two options for supporting this:

1. Using the bull concurrency argument of the process function. (Preferred)

• See:
  • https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queueprocess
  • https://github.com/OptimalBits/bull/tree/develop/docs#sandboxed-processors
• Before we can use this option we need to confirm the relationship between the concurrency number and the number of processes spawned. It may not be feasible if a new child process is spawned for every concurrency automation due to the overhead on the instance.
• If this is successful we can remove the budibase custom threading from automation processing and rely solely on the bull behaviour that we are not spawning additional threads needlessly

2. Update the existing automation processing to enable reading multiple queue messages at once using in-app logic

It it desirable that concurrent processing is used by default in local and not only something we enable in production systems, including debug support.

Before rolling out it would be best to complete: https://github.com/Budibase/budibase/issues/7560 So that we may adjust the concurrency value to find what works best without affecting the http api

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity.